[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 477308928 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003292] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.007884] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009018] pid_max: default: 32768 minimum: 301 [ 0.011029] LSM: Security Framework initializing [ 0.012073] Yama: becoming mindful. [ 0.013048] SELinux: Initializing. [ 0.014109] *** VALIDATE selinux *** [ 0.022360] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027141] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028163] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030032] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031152] *** VALIDATE tmpfs *** [ 0.032479] *** VALIDATE proc *** [ 0.033293] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035261] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038032] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042057] debug: unmapping init [mem 0xffffffff9ae59000-0xffffffff9ae60fff] [ 0.045147] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046693] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047023] ... version: 2 [ 0.048012] ... bit width: 48 [ 0.049017] ... generic registers: 4 [ 0.050016] ... value mask: 0000ffffffffffff [ 0.051017] ... max period: 00007fffffffffff [ 0.052018] ... fixed-purpose events: 3 [ 0.053016] ... event mask: 000000070000000f [ 0.054337] rcu: Hierarchical SRCU implementation. [ 0.056550] smp: Bringing up secondary CPUs ... [ 0.057527] x86: Booting SMP configuration: [ 0.058032] .... node #0, CPUs: #1 #2 #3 [ 0.061737] smp: Brought up 1 node, 4 CPUs [ 0.063014] smpboot: Max logical packages: 1 [ 0.064019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139494] node 0 deferred pages initialised in 73ms [ 0.142114] devtmpfs: initialized [ 0.143255] x86/mm: Memory block size: 128MB [ 0.146872] gcov: version magic: 0x41383552 [ 0.150254] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152282] pinctrl core: initialized pinctrl subsystem [ 0.153308] [ 0.153901] ************************************************************* [ 0.154014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155015] ** ** [ 0.156015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.157015] ** ** [ 0.158018] ** This means that this kernel is built to expose internal ** [ 0.159015] ** IOMMU data structures, which may compromise security on ** [ 0.160017] ** your system. ** [ 0.161015] ** ** [ 0.162015] ** If you see this message and you are not debugging the ** [ 0.163016] ** kernel, report this immediately to your vendor! ** [ 0.164014] ** ** [ 0.165012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166018] ************************************************************* [ 0.167679] NET: Registered protocol family 16 [ 0.168476] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.169063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.170078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.171487] cpuidle: using governor menu [ 0.173323] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.175647] PCI: Using configuration type 1 for base access [ 0.178129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189058] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190055] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192034] cryptd: max_cpu_qlen set to 1000 [ 0.194140] ACPI: Added _OSI(Module Device) [ 0.196012] ACPI: Added _OSI(Processor Device) [ 0.198018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.199024] ACPI: Added _OSI(Processor Aggregator Device) [ 0.204870] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.211146] ACPI: Interpreter enabled [ 0.213072] ACPI: PM: (supports S0 S3 S4 S5) [ 0.214015] ACPI: Using IOAPIC for interrupt routing [ 0.216211] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.219397] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.230942] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.233077] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.236019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.239086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.243416] acpiphp: Slot [2] registered [ 0.245118] acpiphp: Slot [5] registered [ 0.247154] acpiphp: Slot [6] registered [ 0.248081] acpiphp: Slot [7] registered [ 0.249098] acpiphp: Slot [8] registered [ 0.250121] acpiphp: Slot [9] registered [ 0.252132] acpiphp: Slot [10] registered [ 0.253192] acpiphp: Slot [3] registered [ 0.255137] acpiphp: Slot [4] registered [ 0.257141] acpiphp: Slot [11] registered [ 0.259126] acpiphp: Slot [12] registered [ 0.261126] acpiphp: Slot [13] registered [ 0.263110] acpiphp: Slot [14] registered [ 0.265116] acpiphp: Slot [15] registered [ 0.267126] acpiphp: Slot [16] registered [ 0.270108] acpiphp: Slot [17] registered [ 0.272102] acpiphp: Slot [18] registered [ 0.274110] acpiphp: Slot [19] registered [ 0.276105] acpiphp: Slot [20] registered [ 0.278131] acpiphp: Slot [21] registered [ 0.281575] acpiphp: Slot [22] registered [ 0.284120] acpiphp: Slot [23] registered [ 0.286113] acpiphp: Slot [24] registered [ 0.288086] acpiphp: Slot [25] registered [ 0.290118] acpiphp: Slot [26] registered [ 0.292111] acpiphp: Slot [27] registered [ 0.294140] acpiphp: Slot [28] registered [ 0.296116] acpiphp: Slot [29] registered [ 0.298112] acpiphp: Slot [30] registered [ 0.299103] acpiphp: Slot [31] registered [ 0.301067] PCI host bridge to bus 0000:00 [ 0.303022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.305021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.307024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.310023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.314022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.316024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.319013] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.321728] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.325335] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.335982] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.340660] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.343023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.346023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.350018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.352576] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.356086] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.359047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.362810] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.367013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.380016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.385016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.390465] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.399013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.406014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.422025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.431463] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.438016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.443014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.463021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.474834] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.480014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.485015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.504018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.517088] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.523017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.528011] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.545033] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.554028] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.559014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.565017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.580953] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.591136] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.597018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.606016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.623019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.634284] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.637301] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.640454] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.642275] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.645186] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.648122] iommu: Default domain type: Passthrough [ 0.650462] SCSI subsystem initialized [ 0.652132] ACPI: bus type USB registered [ 0.654124] usbcore: registered new interface driver usbfs [ 0.656067] usbcore: registered new interface driver hub [ 0.658083] usbcore: registered new device driver usb [ 0.660274] pps_core: LinuxPPS API ver. 1 registered [ 0.662012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.665064] PTP clock support registered [ 0.668051] EDAC MC: Ver: 3.0.0 [ 0.670148] PCI: Using ACPI for IRQ routing [ 0.671914] NetLabel: Initializing [ 0.672014] NetLabel: domain hash size = 128 [ 0.673009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.674107] NetLabel: unlabeled traffic allowed by default [ 0.675099] vgaarb: loaded [ 0.676283] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.679019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.684355] clocksource: Switched to clocksource kvm-clock [ 0.789690] VFS: Disk quotas dquot_6.6.0 [ 0.791382] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.793714] *** VALIDATE ramfs *** [ 0.795072] *** VALIDATE hugetlbfs *** [ 0.796684] pnp: PnP ACPI init [ 0.799855] pnp: PnP ACPI: found 6 devices [ 0.815460] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.819446] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.821923] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.824247] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.827069] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.829849] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.833017] NET: Registered protocol family 2 [ 0.835630] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.840874] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.844820] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.850333] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.854303] TCP: Hash tables configured (established 65536 bind 65536) [ 0.857390] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.860208] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.862938] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.865963] NET: Registered protocol family 1 [ 0.868027] RPC: Registered named UNIX socket transport module. [ 0.869578] RPC: Registered udp transport module. [ 0.870594] RPC: Registered tcp transport module. [ 0.871605] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.873172] NET: Registered protocol family 44 [ 0.875054] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.877324] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.879499] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.882079] PCI: CLS 0 bytes, default 64 [ 0.884548] Unpacking initramfs... [ 2.287261] debug: unmapping init [mem 0xffff9c92fcc54000-0xffff9c92fffbffff] [ 2.291561] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.296300] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.300635] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.795959] Initialise system trusted keyrings [ 2.797821] Key type blacklist registered [ 2.799279] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.809876] zbud: loaded [ 2.813338] *** VALIDATE nfs *** [ 2.814264] *** VALIDATE nfs4 *** [ 2.816169] pstore: using deflate compression [ 2.819638] Platform Keyring initialized [ 2.922645] NET: Registered protocol family 38 [ 2.924096] Key type asymmetric registered [ 2.925825] Asymmetric key parser 'x509' registered [ 2.927608] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.930476] io scheduler mq-deadline registered [ 2.931863] io scheduler kyber registered [ 2.933510] io scheduler bfq registered [ 2.935299] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.937717] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.940454] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.943302] ACPI: Power Button [PWRF] [ 2.948724] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.955751] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.968164] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.975693] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.996473] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.024509] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.055360] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.060812] Non-volatile memory driver v1.3 [ 3.062700] Linux agpgart interface v0.103 [ 3.096939] virtio_blk virtio1: [vda] 68000 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.100215] vda: detected capacity change from 0 to 34816000 [ 3.117139] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.120594] vdb: detected capacity change from 0 to 1073741824 [ 3.136798] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.139980] vdc: detected capacity change from 0 to 2621440000 [ 3.157152] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.160192] vdd: detected capacity change from 0 to 2621440000 [ 3.179046] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.182242] vde: detected capacity change from 0 to 4294967296 [ 3.198530] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.201865] vdf: detected capacity change from 0 to 4294967296 [ 3.208695] libphy: Fixed MDIO Bus: probed [ 3.215510] usbcore: registered new interface driver usbserial_generic [ 3.217774] usbserial: USB Serial support registered for generic [ 3.219821] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.224127] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.226036] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.228801] mousedev: PS/2 mouse device common for all mice [ 3.232482] rtc_cmos 00:05: RTC can wake from S4 [ 3.235559] rtc_cmos 00:05: registered as rtc0 [ 3.235941] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.237173] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.242902] intel_pstate: CPU model not supported [ 3.246092] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.250092] hid: raw HID events driver (C) Jiri Kosina [ 3.253022] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.253736] usbcore: registered new interface driver usbhid [ 3.253740] usbhid: USB HID core driver [ 3.253870] drop_monitor: Initializing network drop monitor service [ 3.253989] Initializing XFRM netlink socket [ 3.255263] NET: Registered protocol family 10 [ 3.268996] Segment Routing with IPv6 [ 3.270398] NET: Registered protocol family 17 [ 3.271717] mpls_gso: MPLS GSO support [ 3.276875] RAS: Correctable Errors collector initialized. [ 3.279127] AVX version of gcm_enc/dec engaged. [ 3.281146] AES CTR mode by8 optimization enabled [ 3.362694] sched_clock: Marking stable (3362671523, 0)->(4309366335, -946694812) [ 3.366024] registered taskstats version 1 [ 3.368575] Loading compiled-in X.509 certificates [ 3.370578] zswap: loaded using pool lzo/zbud [ 3.395549] Key type big_key registered [ 3.408953] Key type encrypted registered [ 3.410810] ima: No TPM chip found, activating TPM-bypass! [ 3.413102] ima: Allocated hash algorithm: sha1 [ 3.414979] ima: No architecture policies found [ 3.416919] evm: Initialising EVM extended attributes: [ 3.418927] evm: security.selinux [ 3.420571] evm: security.ima [ 3.421717] evm: security.capability [ 3.423279] evm: HMAC attrs: 0x1 [ 3.425821] rtc_cmos 00:05: setting system clock to 2026-04-15 14:58:50 UTC (1776265130) [ 3.432831] debug: unmapping init [mem 0xffffffff9be03000-0xffffffff9bffffff] [ 3.436266] debug: unmapping init [mem 0xffffffff9ab82000-0xffffffff9ae58fff] [ 3.444130] Write protecting the kernel read-only data: 28672k [ 3.447429] debug: unmapping init [mem 0xffffffff99203000-0xffffffff993fffff] [ 3.450432] debug: unmapping init [mem 0xffffffff99b14000-0xffffffff99bfffff] [ 3.483607] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.494219] systemd[1]: Detected virtualization kvm. [ 3.496316] systemd[1]: Detected architecture x86-64. [ 3.498345] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.525934] systemd[1]: No hostname configured. [ 3.527662] systemd[1]: Set hostname to . [ 3.529876] random: systemd: uninitialized urandom read (16 bytes read) [ 3.532330] systemd[1]: Initializing machine ID from random generator. [ 3.574793] random: ln: uninitialized urandom read (6 bytes read) [ 3.659085] random: systemd: uninitialized urandom read (16 bytes read) [ 3.662754] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.672560] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ 3.685556] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.239961] device-mapper: uevent: version 1.0.3 [ 4.242142] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.862368] virtio_net virtio0 ens2: renamed from eth0 [ 4.872508] scsi host0: ata_piix [ 4.892237] random: fast init done [ 4.898619] scsi host1: ata_piix [ 4.902915] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.907079] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.633333] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.938616] random: crng init done [ 9.940365] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.265160] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.469627] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.710114] SELinux: Disabled at runtime. [ 11.769530] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.779672] systemd[1]: Detected virtualization kvm. [ 11.781583] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.280179] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.283162] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.287347] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.290367] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.292973] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.301062] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.321139] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe[ 12.370216] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS . [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.825535] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.177467] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.220100] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.359282] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.369640] EDAC sbridge: Ver: 1.1.2 [ 15.206262] Key type dns_resolver registered [ 15.510572] NFS: Registering the id_resolver key type [ 15.513784] Key type id_resolver registered [ 15.515699] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg616-server login: [ 40.093466] hrtimer: interrupt took 3383051 ns [ 58.930601] libcfs: loading out-of-tree module taints kernel. [ 58.996217] alg: No test for adler32 (adler32-zlib) [ 59.747950] Key type ._llcrypt registered [ 59.752172] Key type .llcrypt registered [ 59.844346] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 77.260738] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 78.595704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 79.338150] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 80.248794] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 80.256279] LNet: Accept secure, port 988 [ 82.048161] Key type lgssc registered [ 83.417786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 97.597575] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 100.162078] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 112.202627] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 120.563366] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 129.879268] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 145.423375] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 157.511590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 157.655538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 158.972482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 159.017292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 159.117039] Lustre: lustre-MDT0000: new disk, initializing [ 159.195983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 159.235427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 163.703611] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 176.216602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 176.303752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 176.377036] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 176.408148] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 176.413123] Lustre: Skipped 1 previous similar message [ 176.414749] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 176.501250] Lustre: lustre-MDT0001: new disk, initializing [ 176.559349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 176.597359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 176.605899] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 180.403581] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 192.869831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 192.938702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 192.946691] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 193.130573] Lustre: lustre-OST0000: new disk, initializing [ 193.134154] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 193.188197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 197.211505] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 201.790611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 201.805216] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 210.498599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 210.558643] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 210.563750] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 210.710614] Lustre: lustre-OST0001: new disk, initializing [ 210.714858] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 210.793369] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 215.902640] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 227.134444] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 233.020498] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 242.130980] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing check_logdir /tmp/testlogs/ [ 246.775487] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing yml_node [ 250.895382] Lustre: DEBUG MARKER: Client: 2.15.8.1 [ 253.968638] Lustre: DEBUG MARKER: MDS: 2.15.8.1 [ 256.503850] Lustre: DEBUG MARKER: OSS: 2.15.8.1 [ 258.245885] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Apr 15 11:03:03 EDT 2026 [ 267.931170] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball 110 [ 269.714427] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 289.248916] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 289.263250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 289.278993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 289.288607] Lustre: Skipped 2 previous similar messages [ 289.310608] Lustre: Skipped 3 previous similar messages [ 293.045238] Lustre: server umount lustre-MDT0000 complete [ 294.382777] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 294.399522] LustreError: Skipped 3 previous similar messages [ 298.170831] LustreError: 6261:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776265425 with bad export cookie 17858485119065103344 [ 298.173424] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 298.189520] LustreError: 6261:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 299.493762] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 299.496535] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 299.497964] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 299.516491] LustreError: Skipped 1 previous similar message [ 299.525067] Lustre: Skipped 2 previous similar messages [ 303.523112] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 303.523773] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 303.553670] Lustre: Skipped 2 previous similar messages [ 303.553946] LustreError: Skipped 1 previous similar message [ 304.583735] Lustre: server umount lustre-MDT0001 complete [ 308.736416] Lustre: server umount lustre-OST0000 complete [ 313.944789] Lustre: server umount lustre-OST0001 complete [ 320.608446] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 328.842734] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 339.507812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 347.052787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 353.831056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 360.761313] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 367.785912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 367.845889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 368.073654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 368.115689] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 368.204397] Lustre: lustre-MDT0000: new disk, initializing [ 368.305270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 368.328632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 373.112795] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 383.377898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 383.457039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 383.523520] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 383.565139] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 383.570332] Lustre: Skipped 1 previous similar message [ 383.637974] Lustre: lustre-MDT0001: new disk, initializing [ 383.728834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 383.760726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 383.772532] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 388.016950] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 396.535414] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 402.266855] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 409.081040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 409.156143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 409.163628] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 409.393823] Lustre: lustre-OST0000: new disk, initializing [ 409.397746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 409.469678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 411.433973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 411.440692] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 412.776629] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 423.346631] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 428.099440] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 428.482431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 434.856346] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 435.218943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 442.340805] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 442.341545] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 442.344197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 442.357600] Lustre: Skipped 1 previous similar message [ 443.668480] Lustre: server umount lustre-OST0000 complete [ 450.528983] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 450.533806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 450.555249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 450.559165] Lustre: Skipped 1 previous similar message [ 454.574833] Lustre: server umount lustre-MDT0000 complete [ 457.702645] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 458.454931] LustreError: 14354:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776265585 with bad export cookie 17858485119065104800 [ 458.461869] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 458.470559] LustreError: 14354:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 463.507383] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 11:06:28 (1776265588) [ 466.021722] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 467.476073] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 11:06:33 (1776265593) [ 474.355437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 474.811285] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 474.824206] LustreError: Skipped 1 previous similar message [ 474.867360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 478.868378] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 485.360748] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.377351] LustreError: Skipped 1 previous similar message [ 487.665363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 493.049971] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 500.894305] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 506.844109] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 512.781952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 517.395961] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 524.149175] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 539.107745] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 539.126588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.149457] Lustre: Skipped 1 previous similar message [ 539.162814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 539.177231] Lustre: Skipped 2 previous similar messages [ 542.622380] Lustre: server umount lustre-OST0000 complete [ 542.628551] Lustre: Skipped 1 previous similar message [ 549.346195] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 549.376378] LustreError: Skipped 1 previous similar message [ 549.381879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 549.405264] Lustre: Skipped 1 previous similar message [ 556.514386] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 556.526064] LustreError: Skipped 1 previous similar message [ 556.641995] LustreError: 18890:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776265683 with bad export cookie 17858485119065105654 [ 556.646056] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 556.659806] LustreError: 18890:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 568.231464] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 570.724929] Key type lgssc unregistered [ 571.064524] LNet: 22582:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 571.075410] LNet: Removed LNI 192.168.206.116@tcp [ 571.793971] Key type .llcrypt unregistered [ 571.797853] Key type ._llcrypt unregistered [ 591.798653] alg: No test for adler32 (adler32-zlib) [ 592.555217] Key type ._llcrypt registered [ 592.565509] Key type .llcrypt registered [ 592.704612] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 593.576504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 593.935638] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 594.160566] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 594.168238] LNet: Accept secure, port 988 [ 595.833231] Key type lgssc registered [ 596.952395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 604.971347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 606.462552] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 606.518791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 610.919325] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 611.810773] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 616.936596] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 619.802988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 620.207736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 624.694621] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 632.464660] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 638.302863] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 644.373739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 644.775066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 649.186456] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 656.045194] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 657.089922] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 666.594524] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 666.600726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 666.616958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 671.720810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 671.728441] Lustre: Skipped 2 previous similar messages [ 672.171157] Lustre: server umount lustre-OST0000 complete [ 676.331925] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 676.339907] LustreError: Skipped 1 previous similar message [ 676.342958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 676.353582] Lustre: Skipped 1 previous similar message [ 676.356980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 681.486336] Lustre: server umount lustre-MDT0000 complete [ 681.956830] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 681.976802] LustreError: Skipped 1 previous similar message [ 685.295511] LustreError: 24025:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776265812 with bad export cookie 7650646857834328564 [ 685.300544] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 685.313064] LustreError: 24025:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 685.745640] Lustre: server umount lustre-MDT0001 complete [ 697.065383] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 699.753122] Key type lgssc unregistered [ 700.006306] LNet: 27705:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.017143] LNet: Removed LNI 192.168.206.116@tcp [ 700.871699] Key type .llcrypt unregistered [ 700.874306] Key type ._llcrypt unregistered [ 720.624145] alg: No test for adler32 (adler32-zlib) [ 721.380556] Key type ._llcrypt registered [ 721.393711] Key type .llcrypt registered [ 721.539590] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 722.666935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 723.304559] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 723.455153] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 723.462096] LNet: Accept secure, port 988 [ 725.112195] Key type lgssc registered [ 726.137386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 734.802808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 736.320595] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 736.394836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 741.368324] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 741.859715] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 746.979096] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 749.510807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 749.809929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 754.327956] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 762.287749] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 767.706245] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 774.200596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 774.556497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 778.903326] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 786.734561] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 786.884067] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 793.926281] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.16@tcp (stopping) [ 796.131669] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 796.139280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 796.156500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 797.157186] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 797.158969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 797.179588] Lustre: Skipped 1 previous similar message [ 797.202694] Lustre: Skipped 2 previous similar messages [ 799.425887] Lustre: server umount lustre-MDT0000 complete [ 802.281757] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 802.295996] LustreError: Skipped 2 previous similar messages [ 803.546509] LustreError: 29149:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776265930 with bad export cookie 8637094974185932038 [ 803.550211] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 803.560789] LustreError: 29149:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 804.197644] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.16@tcp (stopping) [ 804.202765] Lustre: Skipped 1 previous similar message [ 807.393633] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 807.395415] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 807.422148] LustreError: Skipped 1 previous similar message [ 808.418127] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 808.431185] Lustre: Skipped 1 previous similar message [ 809.874418] Lustre: server umount lustre-MDT0001 complete [ 825.459948] Lustre: server umount lustre-OST0000 complete [ 843.669630] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 846.757989] Key type lgssc unregistered [ 847.094775] LNet: 33146:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.108290] LNet: Removed LNI 192.168.206.116@tcp [ 848.052637] Key type .llcrypt unregistered [ 848.061689] Key type ._llcrypt unregistered [ 877.535899] alg: No test for adler32 (adler32-zlib) [ 878.295775] Key type ._llcrypt registered [ 878.298674] Key type .llcrypt registered [ 878.437888] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 879.890297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 880.385778] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 880.608831] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 880.612982] LNet: Accept secure, port 988 [ 882.288138] Key type lgssc registered [ 883.717104] Lustre: Echo OBD driver; http://www.lustre.org/ [ 891.551512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 892.883873] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 892.947118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 897.363718] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 898.027017] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 903.142044] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 905.649042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 906.023391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 910.538768] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 919.183635] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 925.479860] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 932.743231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 933.171121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 937.380675] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 943.409435] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 945.179272] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 951.779283] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 951.787587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 951.797278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 953.831136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.834321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 953.843037] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 953.886274] Lustre: Skipped 2 previous similar messages [ 957.146877] Lustre: server umount lustre-OST0000 complete [ 975.288373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 975.313873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 980.452158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 980.456839] Lustre: Skipped 2 previous similar messages [ 981.604566] Lustre: server umount lustre-MDT0000 complete [ 985.117791] LustreError: 36582:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776266112 with bad export cookie 11439241220678703969 [ 985.124847] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 985.130684] LustreError: 36582:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 991.495276] Lustre: server umount lustre-MDT0001 complete [ 1003.271274] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 1005.630924] Key type lgssc unregistered [ 1005.858693] LNet: 38464:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.865297] LNet: Removed LNI 192.168.206.116@tcp [ 1006.526663] Key type .llcrypt unregistered [ 1006.528778] Key type ._llcrypt unregistered [ 1027.646176] alg: No test for adler32 (adler32-zlib) [ 1028.402644] Key type ._llcrypt registered [ 1028.405995] Key type .llcrypt registered [ 1028.495974] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 1029.275171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1029.851894] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 1030.068286] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 1030.076275] LNet: Accept secure, port 988 [ 1031.768120] Key type lgssc registered [ 1032.684901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1039.232398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1040.690838] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1040.739297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1044.716822] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 1046.001342] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1051.125782] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1052.277050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1052.605534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1056.690428] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 1065.397476] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1071.208081] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1077.302404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1077.642577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1083.136738] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 1091.966140] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 1093.476630] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1099.252524] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1103.719676] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1109.480694] LNetError: 120-3: Refusing connection from 192.168.206.116 for 127.0.0.2@tcp: No matching NI [ 1109.485397] LNetError: 39225:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1109.493034] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1116.375996] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-04-1511h17m20s [ 1117.856916] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1121.101578] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.16@tcp (stopping) [ 1122.786691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.793249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1122.812712] Lustre: Skipped 1 previous similar message [ 1122.814655] LNetError: 120-3: Refusing connection from 192.168.206.116 for 127.0.0.2@tcp: No matching NI [ 1122.814728] LNetError: 39224:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1122.814754] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1122.858182] Lustre: Skipped 2 previous similar messages [ 1126.075466] Lustre: server umount lustre-MDT0000 complete [ 1126.189847] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.206.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1127.928521] LNetError: 120-3: Refusing connection from 192.168.206.116 for 127.0.0.2@tcp: No matching NI [ 1127.946541] LNetError: 39226:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1127.967797] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1130.207682] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1131.299128] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.206.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1131.311957] LustreError: Skipped 3 previous similar messages [ 1133.040064] LNetError: 120-3: Refusing connection from 192.168.206.116 for 127.0.0.2@tcp: No matching NI [ 1133.053627] LNetError: 39227:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1133.061872] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1134.048734] Lustre: 39238:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776266254/real 1776266254] req@000000006d9fb727 x1862550061862336/t0(0) o400->MGC192.168.206.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776266261 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1134.080925] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1136.749642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1139.495473] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.206.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1139.512168] LustreError: Skipped 11 previous similar messages [ 1140.199475] Lustre: Evicted from MGS (at 192.168.206.116@tcp) after server handle changed from 0x49b774a552e409d3 to 0x49b774a552e40e2c [ 1140.227781] Lustre: MGC192.168.206.116@tcp: Connection restored to 192.168.206.116@tcp (at 0@lo) [ 1140.569208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1140.670510] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 1145.510789] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 1145.837194] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1145.842778] LNetError: 120-3: Refusing connection from 192.168.206.116 for 127.0.0.2@tcp: No matching NI [ 1145.867025] LNetError: 39225:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1145.873466] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 1145.883543] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1145.908232] Lustre: Skipped 2 previous similar messages [ 1149.373732] Lustre: DEBUG MARKER: Wait for df (28918) ... [ 1151.121656] Lustre: DEBUG MARKER: done [ 1159.653474] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1159.666042] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.680097] Lustre: Skipped 1 previous similar message [ 1159.684513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1161.194833] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1161.196878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.510704] Lustre: server umount lustre-OST0000 complete [ 1171.427796] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1171.439864] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1171.450325] Lustre: Skipped 1 previous similar message [ 1171.452853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1171.464479] Lustre: Skipped 3 previous similar messages [ 1171.469666] LNetError: 120-3: Refusing connection from 192.168.206.116 for 127.0.0.2@tcp: No matching NI [ 1171.473809] LNetError: 39224:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1171.478473] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1172.740934] Lustre: server umount lustre-MDT0000 complete [ 1176.548801] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1176.580031] LustreError: Skipped 4 previous similar messages [ 1176.804084] LustreError: 39906:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776266303 with bad export cookie 5311842538915892780 [ 1176.811413] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1176.817024] LustreError: 39906:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1177.192903] Lustre: server umount lustre-MDT0001 complete [ 1190.488931] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 1193.431349] Key type lgssc unregistered [ 1193.851769] LNet: 45030:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1193.860791] LNet: Removed LNI 192.168.206.116@tcp [ 1194.899773] Key type .llcrypt unregistered [ 1194.902454] Key type ._llcrypt unregistered [ 1228.045315] alg: No test for adler32 (adler32-zlib) [ 1228.806116] Key type ._llcrypt registered [ 1228.812130] Key type .llcrypt registered [ 1228.976221] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 1242.816892] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 1243.921835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1244.295739] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 1244.567206] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 1244.571261] LNet: Accept secure, port 988 [ 1246.232399] Key type lgssc registered [ 1247.189337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1255.673282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1260.977812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1266.237472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1273.442753] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1280.177789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1280.235946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.427730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1281.454491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1281.537935] Lustre: lustre-MDT0000: new disk, initializing [ 1281.622558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1281.642040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1285.789629] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 1296.151578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1296.217457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1296.278295] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1296.330641] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1296.335597] Lustre: Skipped 1 previous similar message [ 1296.408820] Lustre: lustre-MDT0001: new disk, initializing [ 1296.484364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1296.516363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1296.539161] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1301.790608] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 1313.385762] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1320.247437] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1327.327876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1327.465781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1327.473744] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1327.783435] Lustre: lustre-OST0000: new disk, initializing [ 1327.790420] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1327.906439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1332.267741] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 1335.387108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1335.410985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1341.847900] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1346.876845] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1347.100907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1351.386219] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1351.639125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1356.257625] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1356.258280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1356.266623] LustreError: Skipped 1 previous similar message [ 1356.271752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1356.280065] Lustre: Skipped 1 previous similar message [ 1356.296627] Lustre: Skipped 1 previous similar message [ 1359.711886] Lustre: server umount lustre-OST0000 complete [ 1366.500122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1366.533140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1366.543871] Lustre: Skipped 1 previous similar message [ 1370.563356] Lustre: server umount lustre-MDT0000 complete [ 1371.618942] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1371.653781] LustreError: Skipped 1 previous similar message [ 1372.643782] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.663294] LustreError: Skipped 1 previous similar message [ 1375.110813] LustreError: 50000:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776266502 with bad export cookie 5346427360533862519 [ 1375.113860] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1375.123312] LustreError: 50000:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1375.354770] Lustre: server umount lustre-MDT0001 complete [ 1384.122654] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 11:21:49 (1776266509) [ 1390.778821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1391.324611] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1391.382600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1396.439534] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 1396.705348] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1401.831425] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1407.314692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1407.663966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1412.706945] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 1421.706742] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1427.909898] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1436.183752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1436.631908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1442.002898] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 1451.071934] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1458.458601] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-04-1511h23m03s [ 1460.827859] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1469.255768] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1472.141405] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1472.209064] LustreError: 53695:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1473.545676] LNetError: 120-3: Refusing connection from 192.168.206.116 for 127.0.0.2@tcp: No matching NI [ 1473.557650] LNetError: 47357:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1473.580503] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1492.298076] Lustre: lustre-MDT0000: Client 13e5e440-f789-499f-8c5e-3ba4c484e52d (at 192.168.206.16@tcp) reconnecting [ 1512.304688] LustreError: 53695:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 awake [ 1512.442710] Lustre: lustre-MDT0000: Client 13e5e440-f789-499f-8c5e-3ba4c484e52d (at 192.168.206.16@tcp) reconnecting [ 1514.953422] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1516.958079] Lustre: DEBUG MARKER: done [ 1529.827445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1529.845934] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1529.847803] Lustre: Skipped 1 previous similar message [ 1529.851042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1534.947528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1534.953755] Lustre: Skipped 1 previous similar message [ 1540.064298] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1540.067558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1540.086881] Lustre: Skipped 1 previous similar message [ 1540.238950] Lustre: server umount lustre-OST0000 complete [ 1545.185961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1545.209189] Lustre: Skipped 2 previous similar messages [ 1545.233073] LNetError: 120-3: Refusing connection from 192.168.206.116 for 127.0.0.2@tcp: No matching NI [ 1545.238069] LNetError: 47354:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1545.243386] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1550.307334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1550.321942] Lustre: Skipped 4 previous similar messages [ 1550.329597] LNetError: 120-3: Refusing connection from 192.168.206.116 for 127.0.0.2@tcp: No matching NI [ 1550.336464] LNetError: 47355:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1550.350379] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1550.510374] Lustre: server umount lustre-MDT0000 complete [ 1554.532415] LustreError: 56494:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776266681 with bad export cookie 5346427360533863373 [ 1554.547765] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1568.807317] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 1571.576922] Key type lgssc unregistered [ 1571.918688] LNet: 58120:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.927588] LNet: Removed LNI 192.168.206.116@tcp [ 1572.797433] Key type .llcrypt unregistered [ 1572.802719] Key type ._llcrypt unregistered [ 1608.840843] alg: No test for adler32 (adler32-zlib) [ 1609.613706] Key type ._llcrypt registered [ 1609.615678] Key type .llcrypt registered [ 1609.742434] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 1625.099434] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 1626.303409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1626.830711] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 1627.106451] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 1627.112845] LNet: Accept secure, port 988 [ 1628.776741] Key type lgssc registered [ 1629.966852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1638.450937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1645.047868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1651.653977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1658.232508] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1664.494353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1664.561973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1665.772287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1665.806492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1665.902190] Lustre: lustre-MDT0000: new disk, initializing [ 1665.976964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1665.993643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1670.157672] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 1680.370502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1680.414255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1680.480606] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1680.520395] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1680.524479] Lustre: Skipped 1 previous similar message [ 1680.604329] Lustre: lustre-MDT0001: new disk, initializing [ 1680.685162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1680.723818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1680.738801] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1685.199256] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 1695.755255] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1702.201514] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1709.367458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1709.451505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1709.464511] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1709.774306] Lustre: lustre-OST0000: new disk, initializing [ 1709.778891] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1709.852827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1715.122199] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 1715.798164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1715.805144] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1725.420533] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1729.768501] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1729.976342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1734.634984] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1734.929675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1741.798639] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1741.800842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1741.809000] LustreError: Skipped 1 previous similar message [ 1741.836291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1741.849601] Lustre: Skipped 1 previous similar message [ 1743.105433] Lustre: server umount lustre-OST0000 complete [ 1746.913420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.923798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1746.930075] Lustre: Skipped 2 previous similar messages [ 1746.955201] Lustre: Skipped 1 previous similar message [ 1752.036527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1752.042372] Lustre: Skipped 1 previous similar message [ 1757.157489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1757.170832] Lustre: Skipped 1 previous similar message [ 1761.248431] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1761.352235] Lustre: server umount lustre-MDT0000 complete [ 1762.288687] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1762.304278] LustreError: Skipped 1 previous similar message [ 1765.399438] LustreError: 62237:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776266892 with bad export cookie 7555965026245673576 [ 1765.406698] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1765.420838] LustreError: 62237:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1765.785524] Lustre: server umount lustre-MDT0001 complete [ 1775.044853] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 11:28:20 (1776266900) [ 1776.727802] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1778.342574] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 11:28:23 (1776266903) [ 1790.463580] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 1794.433251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1795.031883] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1795.115881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1796.156593] Lustre: Failing over lustre-MDT0000 [ 1796.161138] LustreError: 67259:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1796.172433] LustreError: 67259:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1796.177910] LustreError: 67259:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 1802.722369] Lustre: 67310:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776266923/real 1776266923] req@00000000f1e0513f x1862550687864320/t0(0) o251->MGC192.168.206.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776266929 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1802.883919] Lustre: server umount lustre-MDT0000 complete [ 1809.769142] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 11:28:55 (1776266935) [ 1816.837325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1817.199390] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1817.274518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1821.413197] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 1822.689643] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1827.811748] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1829.077772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1829.421468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1833.559659] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 1840.460897] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1845.311605] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1851.326640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1851.601966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1855.538796] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 1863.040704] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1872.087992] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1880.553996] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1880.562203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1880.573019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1882.605866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.628650] Lustre: Skipped 1 previous similar message [ 1884.306401] Lustre: server umount lustre-MDT0000 complete [ 1887.720100] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1887.741865] LustreError: Skipped 2 previous similar messages [ 1890.144905] LustreError: 69830:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776267017 with bad export cookie 7555965026245674703 [ 1890.152304] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1890.168135] LustreError: 69830:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1894.886628] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1899.902936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1908.934108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1923.433196] Lustre: Evicted from MGS (at 192.168.206.116@tcp) after server handle changed from 0x68dc2e287ca606cf to 0x68dc2e287ca60e62 [ 1923.444042] Lustre: MGC192.168.206.116@tcp: Connection restored to 192.168.206.116@tcp (at 0@lo) [ 1923.789133] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1923.893755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1929.005607] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1929.030712] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1929.395285] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 1936.224220] Lustre: 60463:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776267019/real 1776267019] req@000000007b63f556 x1862550687883328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776267063 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1937.971358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1938.523987] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:65 [ 1939.616362] Lustre: 60463:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776267022/real 1776267022] req@00000000017533c9 x1862550687883392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776267066 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1941.664146] Lustre: 60464:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776267024/real 1776267024] req@0000000034ccbc92 x1862550687883584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776267068 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1941.691814] Lustre: 60464:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1943.545858] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1943.744056] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 1951.240069] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1951.712164] Lustre: 60462:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776267034/real 1776267034] req@0000000084f60555 x1862550687883776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776267078 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1951.738749] Lustre: 60462:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1957.043893] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1961.954562] Lustre: 60463:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776267045/real 1776267045] req@00000000fa741016 x1862550687884096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776267089 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1961.991060] Lustre: 60463:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1964.003138] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1964.005539] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.018870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1964.027433] Lustre: Skipped 1 previous similar message [ 1964.045349] Lustre: Skipped 5 previous similar messages [ 1964.268691] Lustre: server umount lustre-MDT0000 complete [ 1964.272312] Lustre: Skipped 1 previous similar message [ 1968.395995] LustreError: 67859:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776267095 with bad export cookie 7555965026245676642 [ 1968.407401] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1968.409175] LustreError: 67859:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1976.079032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1980.386978] Lustre: 60465:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776267100/real 1776267100] req@000000002f646a89 x1862550687897088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776267107 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1980.403829] Lustre: 60465:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1985.956299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2001.768303] Lustre: Evicted from MGS (at 192.168.206.116@tcp) after server handle changed from 0x68dc2e287ca60e62 to 0x68dc2e287ca6147b [ 2001.785080] Lustre: MGC192.168.206.116@tcp: Connection restored to (at 0@lo) [ 2002.013613] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2002.033684] LustreError: Skipped 4 previous similar messages [ 2002.111505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2002.117088] Lustre: Skipped 1 previous similar message [ 2002.184247] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 2006.410396] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 2015.500291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2015.991894] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:97 [ 2017.008151] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 2019.862810] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 2027.534579] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2033.866824] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2041.826551] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2041.848447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2041.865517] Lustre: Skipped 2 previous similar messages [ 2041.878429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2055.136519] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2055.341930] Lustre: server umount lustre-MDT0000 complete [ 2055.346410] Lustre: Skipped 1 previous similar message [ 2059.048221] LustreError: 69830:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776267186 with bad export cookie 7555965026245678203 [ 2059.055100] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2059.056013] LustreError: 69830:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2065.334391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2068.744775] Lustre: DEBUG MARKER: files compared the same [ 2070.496320] Lustre: 60463:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776267190/real 1776267190] req@00000000ff6e5a8e x1862550687911744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776267197 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2070.516088] Lustre: 60463:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2089.717579] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 2092.866276] Key type lgssc unregistered [ 2093.253932] LNet: 77932:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2093.264245] LNet: Removed LNI 192.168.206.116@tcp [ 2094.411223] Key type .llcrypt unregistered [ 2094.426251] Key type ._llcrypt unregistered [ 2120.779830] alg: No test for adler32 (adler32-zlib) [ 2121.537749] Key type ._llcrypt registered [ 2121.541621] Key type .llcrypt registered [ 2121.738669] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 2123.546091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2124.250511] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 2124.499640] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 2124.511827] LNet: Accept secure, port 988 [ 2126.264164] Key type lgssc registered [ 2127.323449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2136.679051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2138.221751] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2138.281652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2142.919527] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug +malloc all 8 [ 2143.719338] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2148.841098] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2152.032184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.294346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2156.219645] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug +malloc all 8 [ 2163.640419] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2169.695586] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2175.810476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2176.199369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2180.718978] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug +malloc all 8 [ 2182.386656] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:129 [ 2188.551425] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 2189.199833] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2197.985853] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2197.995545] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2198.022180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2199.010759] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.011189] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2199.011788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2203.514754] Lustre: server umount lustre-OST0000 complete [ 2208.736864] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2208.742053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2208.761599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2208.766880] Lustre: Skipped 1 previous similar message [ 2213.425815] Lustre: server umount lustre-MDT0000 complete [ 2214.369770] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2214.391197] LustreError: Skipped 1 previous similar message [ 2217.981715] LustreError: 79398:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776267345 with bad export cookie 9206060070245902413 [ 2217.989815] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2217.992968] LustreError: 79398:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2218.387258] Lustre: server umount lustre-MDT0001 complete [ 2231.502446] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 2234.291776] Key type lgssc unregistered [ 2234.584216] LNet: 83072:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2234.591695] LNet: Removed LNI 192.168.206.116@tcp [ 2235.437563] Key type .llcrypt unregistered [ 2235.439697] Key type ._llcrypt unregistered [ 2260.134919] alg: No test for adler32 (adler32-zlib) [ 2260.889336] Key type ._llcrypt registered [ 2260.890988] Key type .llcrypt registered [ 2260.997631] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 2262.023535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2262.580197] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 2262.829891] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 2262.835952] LNet: Accept secure, port 988 [ 2264.512134] Key type lgssc registered [ 2265.514505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2276.083850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2293.858198] LustreError: 84531:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.206.116@tcp: failed processing log, type 1: rc = -5 [ 2324.640342] LustreError: 84531:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.206.116@tcp: failed processing log, type 4: rc = -110 [ 2355.489838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2361.440532] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 2370.284149] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2377.922980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2378.195892] Lustre: *** cfs_fail_loc=706, val=0*** [ 2402.732220] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2402.817307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2402.888107] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 2407.913577] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2408.252471] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 2413.031683] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.157289] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.302508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2418.910432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2419.018655] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:161 [ 2424.485974] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 2434.096710] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2441.219686] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2444.769547] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2444.779202] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2444.798909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2445.803083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2445.817747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2450.916441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2450.924195] Lustre: Skipped 1 previous similar message [ 2456.046216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2456.052880] Lustre: Skipped 2 previous similar messages [ 2458.080198] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2458.261604] Lustre: server umount lustre-OST0000 complete [ 2464.747265] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2464.757182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2464.781986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2468.071081] Lustre: server umount lustre-MDT0000 complete [ 2471.395866] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2471.409722] LustreError: Skipped 1 previous similar message [ 2471.659962] LustreError: 84537:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776267598 with bad export cookie 10359172202613974170 [ 2471.661566] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2471.666678] LustreError: 84537:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2471.956822] Lustre: server umount lustre-MDT0001 complete [ 2483.352778] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 2485.992947] Key type lgssc unregistered [ 2486.243467] LNet: 88276:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2486.259412] LNet: Removed LNI 192.168.206.116@tcp [ 2486.957879] Key type .llcrypt unregistered [ 2486.961325] Key type ._llcrypt unregistered [ 2510.673548] alg: No test for adler32 (adler32-zlib) [ 2511.434791] Key type ._llcrypt registered [ 2511.436940] Key type .llcrypt registered [ 2511.587814] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 2512.743504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2513.150842] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 2513.373333] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 2513.377285] LNet: Accept secure, port 988 [ 2515.064193] Key type lgssc registered [ 2516.169771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2524.162181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2529.725184] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 2537.183390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2537.600159] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2537.730512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2542.411607] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 2543.074067] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.091717] LustreError: Skipped 2 previous similar messages [ 2548.201790] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2548.227814] LustreError: Skipped 1 previous similar message [ 2551.572131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2551.935432] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2551.955258] LustreError: Skipped 2 previous similar messages [ 2551.983846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2553.009144] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:193 [ 2557.087305] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 2558.445716] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.465802] LustreError: Skipped 5 previous similar messages [ 2566.609139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2566.673547] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:193 [ 2572.079909] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 2587.107953] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2587.109402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2587.122165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2587.127816] Lustre: Skipped 1 previous similar message [ 2590.968161] Lustre: server umount lustre-OST0000 complete [ 2597.351617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2597.354201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2597.354336] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2597.370488] Lustre: Skipped 1 previous similar message [ 2597.389384] Lustre: Skipped 1 previous similar message [ 2601.228668] Lustre: server umount lustre-MDT0000 complete [ 2602.465682] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2602.473692] LustreError: Skipped 7 previous similar messages [ 2604.984749] Lustre: server umount lustre-MDT0001 complete [ 2614.752450] Lustre: 93148:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776267735/real 1776267735] req@00000000fb646641 x1862551616895104/t0(0) o251->MGC192.168.206.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776267741 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2614.805355] Lustre: server umount MGS complete [ 2626.560833] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 11:42:31 (1776267751) [ 2662.450204] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 2673.457202] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 2683.850435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2690.655373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2697.694474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2705.583969] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2713.116892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2713.189309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2718.264384] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 2728.642319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2728.713364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2728.890312] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2728.924121] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2729.026974] Lustre: lustre-MDT0001: new disk, initializing [ 2729.132449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2729.154025] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2729.169090] LustreError: 97867:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osd get update log failed: rc = -115 [ 2733.963813] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 2744.975429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2745.043758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2745.054137] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2745.272391] Lustre: lustre-OST0000: new disk, initializing [ 2745.278656] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2745.384121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2750.777508] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 2751.522148] Lustre: 98963:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@000000005ca0234d x1862551616899776/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1776267890 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2761.744881] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2769.530306] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2769.561139] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2769.650386] Lustre: lustre-MDT0000: new disk, initializing [ 2769.817982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2769.855412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2770.912936] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2770.916506] Lustre: 99717:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@0000000041305f5a x1862551616903488/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1776267909 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2774.849833] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 2784.239433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2784.262284] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2787.314511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2787.347798] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2807.779454] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2807.797353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2807.826724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2807.835815] Lustre: Skipped 1 previous similar message [ 2810.852961] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2810.859341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2810.873566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2810.919789] Lustre: Skipped 1 previous similar message [ 2815.971332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2815.984325] Lustre: Skipped 1 previous similar message [ 2821.093806] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2821.315441] Lustre: server umount lustre-OST0000 complete [ 2826.209705] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2826.228268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2826.237109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2826.242591] Lustre: Skipped 1 previous similar message [ 2826.250923] Lustre: Skipped 3 previous similar messages [ 2832.444852] Lustre: server umount lustre-MDT0000 complete [ 2836.484403] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2836.503235] LustreError: Skipped 3 previous similar messages [ 2854.368246] Lustre: 100912:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776267975/real 1776267975] req@00000000baa52448 x1862551616915392/t0(0) o251->MGC192.168.206.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776267981 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2854.405734] Lustre: server umount MGS complete [ 2854.408240] Lustre: Skipped 1 previous similar message [ 2867.857588] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 11:46:32 (1776267992) [ 2892.497233] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 2895.993841] Key type lgssc unregistered [ 2896.231455] LNet: 102325:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2896.237817] LNet: Removed LNI 192.168.206.116@tcp [ 2897.175755] Key type .llcrypt unregistered [ 2897.178352] Key type ._llcrypt unregistered [ 2915.643838] alg: No test for adler32 (adler32-zlib) [ 2916.410520] Key type ._llcrypt registered [ 2916.418839] Key type .llcrypt registered [ 2916.636701] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 2917.717576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2918.385273] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 2918.804400] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 2918.815119] LNet: Accept secure, port 988 [ 2920.560127] Key type lgssc registered [ 2922.022335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2930.645544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2932.218139] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2932.317214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2937.318391] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2938.063983] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 2942.438624] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2947.560112] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2948.323721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2948.725566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2954.555708] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 2964.636771] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2972.705336] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2980.439172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2980.809157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2982.849069] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2985.755606] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 2995.333611] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3008.481629] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3008.482827] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3008.495397] LustreError: Skipped 1 previous similar message [ 3008.500189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3008.516601] Lustre: Skipped 1 previous similar message [ 3008.534660] Lustre: Skipped 1 previous similar message [ 3013.607248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3013.616672] Lustre: Skipped 1 previous similar message [ 3014.720613] Lustre: server umount lustre-OST0000 complete [ 3020.259747] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3020.271560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3020.288822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3023.843174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3023.845144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3023.872201] Lustre: Skipped 1 previous similar message [ 3025.496797] Lustre: server umount lustre-MDT0000 complete [ 3028.964761] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3028.989986] LustreError: Skipped 1 previous similar message [ 3030.384206] LustreError: 103592:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776268157 with bad export cookie 4285080166570592912 [ 3030.404052] LustreError: 103592:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3030.410097] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3030.816424] Lustre: server umount lustre-MDT0001 complete [ 3045.152425] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 3048.685897] Key type lgssc unregistered [ 3049.064955] LNet: 107287:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3049.075520] LNet: Removed LNI 192.168.206.116@tcp [ 3050.257234] Key type .llcrypt unregistered [ 3050.264411] Key type ._llcrypt unregistered [ 3070.004109] alg: No test for adler32 (adler32-zlib) [ 3070.764474] Key type ._llcrypt registered [ 3070.777578] Key type .llcrypt registered [ 3070.923573] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 3072.234226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3073.158977] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 3073.473597] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 3073.487473] LNet: Accept secure, port 988 [ 3075.208319] Key type lgssc registered [ 3076.509708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3083.834126] LustreError: 108435:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 3083.856472] LustreError: 108436:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 3083.874770] LustreError: 108435:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4972 [ 3083.887371] LustreError: 108435:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3083.907673] LustreError: 108435:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3083.924205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3085.490384] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3085.567390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3090.913573] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3095.077293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3095.409330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3100.561253] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 3106.778312] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing lsmod [ 3110.068077] LustreError: 109740:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 3110.112816] LustreError: 109746:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 3110.118658] LustreError: 109740:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4972 [ 3110.125539] LustreError: 109740:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3110.133438] LustreError: 109740:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3110.155506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3110.673764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3116.002911] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3116.027144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3116.047762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3117.034680] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3117.035661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3117.050313] Lustre: Skipped 1 previous similar message [ 3117.066563] Lustre: Skipped 2 previous similar messages [ 3122.150022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3122.162920] Lustre: Skipped 2 previous similar messages [ 3127.268793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3127.277165] Lustre: Skipped 2 previous similar messages [ 3128.288153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3128.436024] Lustre: server umount lustre-MDT0000 complete [ 3132.392465] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3132.421463] LustreError: Skipped 2 previous similar messages [ 3132.610907] LustreError: 108449:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776268259 with bad export cookie 6175822044893513621 [ 3132.638364] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3132.963612] Lustre: server umount lustre-MDT0001 complete [ 3143.139690] Lustre: 110338:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776268264/real 1776268264] req@000000000021bbb0 x1862552204094528/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776268270 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3143.469739] Lustre: server umount lustre-OST0000 complete [ 3152.809804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3153.361618] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3153.417635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3158.503186] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3158.594152] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 3168.753328] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3168.787250] LustreError: Skipped 1 previous similar message [ 3169.431848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3169.777688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3174.934935] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 3183.344888] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3190.820719] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3199.318953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3199.652489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3203.775189] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 3205.536459] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 3214.207369] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3227.116609] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3227.126457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.134175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3229.669971] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3229.679580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3231.164762] Lustre: server umount lustre-OST0000 complete [ 3234.785121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3239.905848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3239.908180] Lustre: Skipped 4 previous similar messages [ 3249.120413] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3249.277134] Lustre: server umount lustre-MDT0000 complete [ 3250.151791] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3250.177826] LustreError: Skipped 1 previous similar message [ 3253.634409] LustreError: 110794:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776268380 with bad export cookie 6175822044893514321 [ 3253.645370] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3253.652391] LustreError: 110794:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3266.611596] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 3269.233585] Key type lgssc unregistered [ 3269.455532] LNet: 114475:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3269.471279] LNet: Removed LNI 192.168.206.116@tcp [ 3270.305718] Key type .llcrypt unregistered [ 3270.308810] Key type ._llcrypt unregistered [ 3293.645308] alg: No test for adler32 (adler32-zlib) [ 3294.406253] Key type ._llcrypt registered [ 3294.411536] Key type .llcrypt registered [ 3294.593459] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 3296.067748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3296.469405] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 3296.706939] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 3296.713098] LNet: Accept secure, port 988 [ 3298.384136] Key type lgssc registered [ 3299.298069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3308.547104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3310.388235] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3310.469164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3315.699832] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3316.250954] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 3320.822965] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3325.923523] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3327.332254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3327.799298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3332.652576] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 3341.628046] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3347.182631] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3353.703396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3354.126078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3358.803304] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 3360.319378] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 3366.996574] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3372.248342] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 3376.150331] Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client [ 3380.705941] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3380.713623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3380.722369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3380.738481] Lustre: Skipped 1 previous similar message [ 3385.832345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3385.840667] Lustre: Skipped 2 previous similar messages [ 3386.456123] Lustre: server umount lustre-OST0000 complete [ 3390.947965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3390.956940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3390.961263] Lustre: Skipped 1 previous similar message [ 3390.979321] Lustre: Skipped 1 previous similar message [ 3396.065721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3396.078759] Lustre: Skipped 1 previous similar message [ 3401.189247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3401.200494] Lustre: Skipped 1 previous similar message [ 3404.256970] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3404.403700] Lustre: server umount lustre-MDT0000 complete [ 3406.310577] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3406.330604] LustreError: Skipped 1 previous similar message [ 3407.915610] LustreError: 118541:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776268534 with bad export cookie 2979336302317124483 [ 3407.924119] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3407.925540] LustreError: 118541:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3408.206110] Lustre: server umount lustre-MDT0001 complete [ 3418.543545] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 3421.422457] Key type lgssc unregistered [ 3421.843126] LNet: 119700:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3421.856823] LNet: Removed LNI 192.168.206.116@tcp [ 3422.620151] Key type .llcrypt unregistered [ 3422.623776] Key type ._llcrypt unregistered [ 3441.271866] alg: No test for adler32 (adler32-zlib) [ 3442.032962] Key type ._llcrypt registered [ 3442.041570] Key type .llcrypt registered [ 3442.196086] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 3443.212184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3444.033779] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 3444.317031] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 3444.321893] LNet: Accept secure, port 988 [ 3446.032323] Key type lgssc registered [ 3447.364547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3455.854478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3457.343176] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3457.412874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3462.650088] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3462.998126] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 3467.754807] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3472.545943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3472.866738] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3473.017674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3477.927741] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 3487.656557] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3494.712896] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3502.657942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3503.149836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3507.255461] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 3508.872623] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 3516.755169] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3529.202107] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3529.207621] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3529.235738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3532.770349] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3532.775851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3532.776632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3534.389755] Lustre: server umount lustre-OST0000 complete [ 3539.431282] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3539.435580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3539.447776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3539.450135] Lustre: Skipped 1 previous similar message [ 3543.019379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3548.130082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3548.137933] Lustre: Skipped 3 previous similar messages [ 3553.249100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3553.394407] Lustre: server umount lustre-MDT0000 complete [ 3556.799228] LustreError: 120969:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776268683 with bad export cookie 12371487597404858045 [ 3556.803659] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3556.809884] LustreError: 120969:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3557.130834] Lustre: server umount lustre-MDT0001 complete [ 3568.037246] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 3570.055463] Key type lgssc unregistered [ 3570.303385] LNet: 124650:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3570.309037] LNet: Removed LNI 192.168.206.116@tcp [ 3570.896542] Key type .llcrypt unregistered [ 3570.899403] Key type ._llcrypt unregistered [ 3591.229357] alg: No test for adler32 (adler32-zlib) [ 3592.015192] Key type ._llcrypt registered [ 3592.018555] Key type .llcrypt registered [ 3592.167632] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 3593.155101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3593.855502] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 3594.172079] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 3594.182839] LNet: Accept secure, port 988 [ 3595.913971] Key type lgssc registered [ 3597.157135] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3604.988693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3606.540833] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3606.636053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3610.962886] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 3611.624021] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3616.746647] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3620.230293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3620.816127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3626.390645] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 3635.588428] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3643.740983] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3651.249719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3651.755561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3656.812836] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 3656.899925] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 3665.058750] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3675.065254] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 3679.363813] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 3679.371416] Lustre: Skipped 2 previous similar messages [ 3682.892557] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3682.936075] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3684.297839] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 3684.310448] Lustre: Skipped 2 previous similar messages [ 3692.129634] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3692.134988] Lustre: Skipped 1 previous similar message [ 3697.490232] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 3697.499019] Lustre: Skipped 2 previous similar messages [ 3702.373129] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.16@tcp 192.168.0.[2,10]@tcp [ 3708.385823] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3708.397833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3708.421956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3710.355345] Lustre: server umount lustre-OST0000 complete [ 3716.577750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3716.580719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3716.598777] Lustre: Skipped 1 previous similar message [ 3716.614908] Lustre: Skipped 1 previous similar message [ 3718.624165] Lustre: 125424:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776268838/real 1776268838] req@00000000c1234730 x1862552750420224/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776268845 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3718.661291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3718.682931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3718.692553] Lustre: Skipped 1 previous similar message [ 3720.475676] Lustre: server umount lustre-MDT0000 complete [ 3723.616299] Lustre: 125425:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776268843/real 1776268843] req@00000000ee07c12c x1862552750421120/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776268850 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3723.630083] Lustre: 125425:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3723.745783] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3723.764357] LustreError: Skipped 1 previous similar message [ 3724.673133] LustreError: 126093:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776268851 with bad export cookie 1941728749253265221 [ 3724.673975] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3724.691545] LustreError: 126093:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3725.095647] Lustre: server umount lustre-MDT0001 complete [ 3737.367757] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 3740.243620] Key type lgssc unregistered [ 3740.543831] LNet: 130938:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3740.563138] LNet: Removed LNI 192.168.206.116@tcp [ 3741.375533] Key type .llcrypt unregistered [ 3741.382876] Key type ._llcrypt unregistered [ 3763.906491] alg: No test for adler32 (adler32-zlib) [ 3764.663495] Key type ._llcrypt registered [ 3764.666795] Key type .llcrypt registered [ 3764.860545] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 3765.959995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3766.427371] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 3766.659756] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 3766.664905] LNet: Accept secure, port 988 [ 3768.368334] Key type lgssc registered [ 3769.616695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3777.321647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3778.839134] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3778.870173] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3778.888778] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.206.16@tcp 192.168.0.[2,10]@tcp [ 3778.910086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3782.998562] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 3784.166905] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3789.283154] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3791.548716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3791.796415] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3791.817162] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.16@tcp 192.168.0.[2,10]@tcp [ 3791.872832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3796.837963] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 3803.626538] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3808.783615] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3813.940502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3814.221365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3818.633892] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 3827.958216] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3829.562559] Lustre: lustre-OST0000: deleting orphan objects from 0x0:167 to 0x0:193 [ 3843.043336] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3843.057125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3843.073842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3845.089717] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3845.090991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3845.105621] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3847.923735] Lustre: server umount lustre-OST0000 complete [ 3853.281210] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3853.287838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3853.313531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3853.319863] Lustre: Skipped 1 previous similar message [ 3855.330931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3855.331776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3855.348138] Lustre: Skipped 1 previous similar message [ 3860.450109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3860.455028] Lustre: Skipped 1 previous similar message [ 3866.080216] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3866.209131] Lustre: server umount lustre-MDT0000 complete [ 3866.599728] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3866.631077] LustreError: Skipped 1 previous similar message [ 3869.536121] LustreError: 132527:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776268996 with bad export cookie 9109257751164204877 [ 3869.548056] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3869.929292] Lustre: server umount lustre-MDT0001 complete [ 3880.463786] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 3883.264994] Key type lgssc unregistered [ 3883.548797] LNet: 136256:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3883.560261] LNet: Removed LNI 192.168.206.116@tcp [ 3884.208774] Key type .llcrypt unregistered [ 3884.214209] Key type ._llcrypt unregistered [ 3905.324377] alg: No test for adler32 (adler32-zlib) [ 3906.078522] Key type ._llcrypt registered [ 3906.082276] Key type .llcrypt registered [ 3906.154777] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 3916.810390] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 3917.747639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3918.190977] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 3918.390182] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 3918.397594] LNet: Accept secure, port 988 [ 3920.081975] Key type lgssc registered [ 3921.124950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3928.679781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3934.293893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3939.660466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3945.380304] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3951.641215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3951.682362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3952.888544] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3952.927561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3953.030140] Lustre: lustre-MDT0000: new disk, initializing [ 3953.161134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3953.195410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3956.972489] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 3966.520636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3966.614166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3966.696845] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3966.736271] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3966.740722] Lustre: Skipped 1 previous similar message [ 3966.857853] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3966.880407] Lustre: lustre-MDT0001: new disk, initializing [ 3966.997342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3967.063141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3967.069946] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3971.631916] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 3979.734830] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3984.484156] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3990.640871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3990.727300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3990.735874] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3991.018811] Lustre: lustre-OST0000: new disk, initializing [ 3991.022473] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3991.114269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3995.543721] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 3999.257298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3999.266956] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4004.034942] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4007.751530] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4007.932400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4011.880283] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4012.105217] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4014.564601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4014.563415] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4014.587873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4014.589128] LustreError: Skipped 1 previous similar message [ 4014.601831] Lustre: Skipped 1 previous similar message [ 4019.684582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4019.693437] Lustre: Skipped 1 previous similar message [ 4024.804571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4024.814927] Lustre: Skipped 1 previous similar message [ 4027.874183] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4028.094719] Lustre: server umount lustre-OST0000 complete [ 4033.505514] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4033.519524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4033.542094] Lustre: Skipped 1 previous similar message [ 4033.544655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4035.043881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4045.285337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4045.294130] Lustre: Skipped 5 previous similar messages [ 4046.306525] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4046.420123] Lustre: server umount lustre-MDT0000 complete [ 4049.936399] LustreError: 139769:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776269177 with bad export cookie 44498060572417467 [ 4049.940386] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4049.953044] LustreError: 139769:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4050.289080] Lustre: server umount lustre-MDT0001 complete [ 4057.532171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4058.022856] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4058.086937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4062.302984] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 4063.206576] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.331524] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4069.858036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4070.271368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4074.188502] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 4081.619329] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4088.465611] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4095.285167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4095.696114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4100.217260] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 4108.742462] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4114.359583] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4114.611408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4118.735888] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4118.921359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4127.014219] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4127.111558] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4127.118334] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4127.272652] Lustre: lustre-OST0001: new disk, initializing [ 4127.276640] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 4127.384487] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4131.650935] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 4132.948184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4132.955144] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4139.255099] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 4139.466772] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4143.398596] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 4143.565456] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4149.214143] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff936346563800.ost_server_uuid 40 [ 4150.664315] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff936346563800.ost_server_uuid in FULL state after 0 sec [ 4158.953076] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4158.956148] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4158.958434] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4158.964598] Lustre: Skipped 1 previous similar message [ 4158.979577] LustreError: Skipped 1 previous similar message [ 4164.865635] Lustre: server umount lustre-OST0001 complete [ 4169.192495] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4169.193900] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4169.209716] LustreError: Skipped 1 previous similar message [ 4169.231262] Lustre: Skipped 1 previous similar message [ 4174.355122] Lustre: server umount lustre-OST0000 complete [ 4179.429037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4179.447052] Lustre: Skipped 1 previous similar message [ 4192.225541] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4192.341373] Lustre: server umount lustre-MDT0000 complete [ 4195.812697] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4195.835763] LustreError: Skipped 1 previous similar message [ 4196.169382] LustreError: 144116:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776269323 with bad export cookie 44498060572418321 [ 4196.196464] LustreError: 144116:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4196.199606] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4220.289510] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 4222.821441] Key type lgssc unregistered [ 4223.067105] LNet: 150789:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4223.081453] LNet: Removed LNI 192.168.206.116@tcp [ 4223.845672] Key type .llcrypt unregistered [ 4223.848967] Key type ._llcrypt unregistered [ 4259.135754] alg: No test for adler32 (adler32-zlib) [ 4259.901160] Key type ._llcrypt registered [ 4259.903277] Key type .llcrypt registered [ 4260.090119] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 4275.104769] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 4277.611694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4278.489796] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 4278.890544] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 4278.894796] LNet: Accept secure, port 988 [ 4280.624146] Key type lgssc registered [ 4281.657742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4292.128883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4298.619932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4305.679444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4312.434538] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4319.711471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4319.788804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4321.071629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4321.118985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4321.219826] Lustre: lustre-MDT0000: new disk, initializing [ 4321.292115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4321.326623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4325.601492] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 4336.183227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4336.258434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4336.344292] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4336.396660] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4336.401410] Lustre: Skipped 1 previous similar message [ 4336.495449] Lustre: lustre-MDT0001: new disk, initializing [ 4336.583764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4336.645715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4336.661427] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4340.647843] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 4349.363452] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4354.864052] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4361.088981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4361.154804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4361.161602] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4361.331187] Lustre: lustre-OST0000: new disk, initializing [ 4361.334647] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4361.416044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4364.910434] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 4370.537076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4370.552540] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4374.801600] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4380.488601] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4380.706706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4384.383349] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4384.565889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4391.397167] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4391.406952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4391.408257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4391.439053] Lustre: Skipped 1 previous similar message [ 4391.462643] Lustre: Skipped 1 previous similar message [ 4392.488958] Lustre: server umount lustre-OST0000 complete [ 4398.058601] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4398.072995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4398.103180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4401.633842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4401.637716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4403.322377] Lustre: server umount lustre-MDT0000 complete [ 4406.753485] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4406.770992] LustreError: Skipped 1 previous similar message [ 4406.882556] LustreError: 154901:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776269533 with bad export cookie 9903080272956360060 [ 4406.889740] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4406.902721] LustreError: 154901:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4407.151414] Lustre: server umount lustre-MDT0001 complete [ 4415.564909] Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ========================================================== 12:12:20 (1776269540) [ 4432.279717] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 4440.888807] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 4451.278244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4458.486476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4464.859658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4470.765295] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4476.636372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4476.698261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4476.909396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4476.932150] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4477.022802] Lustre: lustre-MDT0000: new disk, initializing [ 4477.185216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4477.215933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4481.440811] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 4492.311693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4492.392791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4492.504318] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4492.599394] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4492.604590] Lustre: Skipped 1 previous similar message [ 4492.731188] Lustre: lustre-MDT0001: new disk, initializing [ 4492.839154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4492.879918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4492.898261] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4498.288884] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 4507.703572] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4513.732140] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4521.737621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4521.792337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4521.798115] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4522.034451] Lustre: lustre-OST0000: new disk, initializing [ 4522.038250] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4522.108511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4526.968599] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 4527.717870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4527.729021] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4536.510369] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4541.062114] Lustre: Failing over lustre-OST0000 [ 4541.209431] Lustre: server umount lustre-OST0000 complete [ 4550.624229] Lustre: 153128:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776269670/real 1776269670] req@00000000120e3e3f x1862553467651904/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1776269677 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 4550.625694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4550.698141] Lustre: 153128:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4550.764283] Lustre: Skipped 1 previous similar message [ 4555.168160] Lustre: 153128:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776269675/real 1776269675] req@000000005b4531ef x1862553467652736/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776269682 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4555.208837] Lustre: 153128:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4562.391213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4562.750189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4562.768777] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4567.246204] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 4567.854991] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4567.937462] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4567.939492] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 4567.968950] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 4571.958016] Lustre: Failing over lustre-MDT0000 [ 4572.098393] Lustre: server umount lustre-MDT0000 complete [ 4573.022706] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.206.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4573.171880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4573.201821] Lustre: Skipped 1 previous similar message [ 4578.115226] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.206.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4578.131922] LustreError: Skipped 3 previous similar messages [ 4580.188260] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.206.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4580.216957] LustreError: Skipped 3 previous similar messages [ 4580.320162] Lustre: 153127:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776269700/real 1776269700] req@00000000120e3e3f x1862553467658304/t0(0) o400->MGC192.168.206.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776269707 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4580.361495] Lustre: 153127:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4580.369196] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4585.257151] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.206.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4585.278376] LustreError: Skipped 3 previous similar messages [ 4593.701603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4596.519575] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.206.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4596.531187] LustreError: Skipped 11 previous similar messages [ 4597.748823] Lustre: Evicted from MGS (at 192.168.206.116@tcp) after server handle changed from 0x896ecf0a1bbe08cb to 0x896ecf0a1bbe0e89 [ 4597.767413] Lustre: MGC192.168.206.116@tcp: Connection restored to (at 0@lo) [ 4597.780456] Lustre: Skipped 1 previous similar message [ 4598.122856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4598.185830] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4602.664432] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4603.367341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 4603.407905] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4603.470986] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 4603.606613] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 4613.610312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4613.617383] Lustre: Skipped 2 previous similar messages [ 4613.617591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4613.622670] Lustre: Skipped 2 previous similar messages [ 4618.535581] Lustre: server umount lustre-OST0000 complete [ 4623.848343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4623.850259] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4628.967877] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4628.984865] LustreError: Skipped 7 previous similar messages [ 4632.862199] LustreError: 163983:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776269759 with bad export cookie 9903080272956362377 [ 4632.865175] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4632.885521] LustreError: 163983:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4647.403455] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 4650.692503] Key type lgssc unregistered [ 4651.093337] LNet: 167755:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4651.111816] LNet: Removed LNI 192.168.206.116@tcp [ 4652.290265] Key type .llcrypt unregistered [ 4652.294314] Key type ._llcrypt unregistered [ 4681.049811] alg: No test for adler32 (adler32-zlib) [ 4681.804523] Key type ._llcrypt registered [ 4681.807024] Key type .llcrypt registered [ 4681.895987] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 4683.288705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4683.957905] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 4684.170708] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 4684.176604] LNet: Accept secure, port 988 [ 4685.842816] Key type lgssc registered [ 4687.110858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4695.627350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4697.290258] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4697.370243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4702.693970] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4702.826749] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 4707.821037] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4711.814188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4712.175201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4717.063876] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 4725.780510] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4732.618197] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4739.291601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4739.709454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4744.366816] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 4748.931034] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:97 [ 4752.830489] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4860.285286] Lustre: lustre-MDT0000: Client c9dd091e-f93d-497f-ac59-7f948bbf225d (at 192.168.206.16@tcp) reconnecting [ 5122.396636] Lustre: lustre-MDT0000: Client c9dd091e-f93d-497f-ac59-7f948bbf225d (at 192.168.206.16@tcp) reconnecting [ 5192.000658] Lustre: lustre-MDT0000: Client c9dd091e-f93d-497f-ac59-7f948bbf225d (at 192.168.206.16@tcp) reconnecting [ 5199.652889] Lustre: Failing over lustre-MDT0000 [ 5199.842067] Lustre: server umount lustre-MDT0000 complete [ 5202.245275] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.206.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5202.917997] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5202.927936] Lustre: Skipped 2 previous similar messages [ 5207.338146] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.206.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5207.354730] LustreError: Skipped 3 previous similar messages [ 5209.056159] Lustre: 168553:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776270329/real 1776270329] req@00000000c719b628 x1862553893710016/t0(0) o400->MGC192.168.206.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776270336 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5209.089131] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5215.526576] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.206.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5215.545190] LustreError: Skipped 14 previous similar messages [ 5217.639242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5226.476077] Lustre: Evicted from MGS (at 192.168.206.116@tcp) after server handle changed from 0x42d0b420fa265671 to 0x42d0b420fa28c292 [ 5226.491373] LustreError: 172514:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.206.116@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000008cbc4033) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 5226.506143] Lustre: MGC192.168.206.116@tcp: Connection restored to (at 0@lo) [ 5226.761904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5226.825412] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5231.384576] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 5231.915659] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5232.127447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 5232.165641] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5232.204878] Lustre: lustre-OST0000: deleting orphan objects from 0x0:100 to 0x0:129 [ 5239.904565] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5241.576713] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5257.705542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5257.714884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5257.721113] Lustre: Skipped 1 previous similar message [ 5257.723697] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5257.741191] Lustre: Skipped 2 previous similar messages [ 5262.824431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5262.831296] Lustre: Skipped 2 previous similar messages [ 5263.273369] Lustre: server umount lustre-MDT0000 complete [ 5266.899179] LustreError: 169221:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776270393 with bad export cookie 4814546055387988626 [ 5266.900733] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5266.917258] LustreError: 169221:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5267.937731] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5267.940751] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5267.949983] LustreError: Skipped 11 previous similar messages [ 5267.969293] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5271.972259] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5278.180173] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5278.185067] Lustre: Skipped 1 previous similar message [ 5281.248520] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5281.538202] Lustre: server umount lustre-MDT0001 complete [ 5291.416790] Lustre: server umount lustre-OST0000 complete [ 5300.099954] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 5308.562038] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 5317.120806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5322.803727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5328.758773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5334.282552] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5341.205045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5341.265949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5341.472794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5341.492517] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5341.560129] Lustre: lustre-MDT0000: new disk, initializing [ 5341.669241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5341.688473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5345.512295] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 5355.407453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5355.515058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5355.591866] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5355.655863] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5355.664644] Lustre: Skipped 1 previous similar message [ 5355.752410] Lustre: lustre-MDT0001: new disk, initializing [ 5355.863824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5355.900172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5355.920420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5360.185555] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 5369.883940] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5375.195088] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5381.014190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5381.067606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5381.071435] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5381.224939] Lustre: lustre-OST0000: new disk, initializing [ 5381.229872] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5381.334794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5385.284276] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 5388.862821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5388.872492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5392.991835] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5396.983662] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5397.175776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5400.738772] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5400.883313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5404.647186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5404.647768] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5404.649267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5404.664803] Lustre: Skipped 1 previous similar message [ 5408.416146] Lustre: server umount lustre-OST0000 complete [ 5412.321763] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5412.331823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5417.590378] Lustre: server umount lustre-MDT0000 complete [ 5420.001981] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5420.016135] LustreError: Skipped 4 previous similar messages [ 5420.167545] LustreError: 176158:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776270547 with bad export cookie 4814546055387989347 [ 5420.170230] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5420.179225] LustreError: 176158:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5427.252511] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:29:12 (1776270552) [ 5439.835520] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 5445.921858] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 5452.798613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5458.713286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5463.335516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5467.775884] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5472.568970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5472.608022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5472.783161] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5472.816802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5472.882557] Lustre: lustre-MDT0000: new disk, initializing [ 5472.965701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5472.979675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5477.004196] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 5486.439806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5486.517741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5486.607209] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5486.661995] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5486.671027] Lustre: Skipped 1 previous similar message [ 5486.757328] Lustre: lustre-MDT0001: new disk, initializing [ 5486.863269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5486.874526] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5490.615751] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 5499.746669] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5504.749706] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5510.357948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5510.455855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5510.466230] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5510.749494] Lustre: lustre-OST0000: new disk, initializing [ 5510.753950] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5514.611945] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 5518.884904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5518.893112] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5523.002812] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5529.064414] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5529.065717] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5529.071427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5529.071436] Lustre: Skipped 4 previous similar messages [ 5529.087263] Lustre: Skipped 2 previous similar messages [ 5529.093283] LustreError: Skipped 1 previous similar message [ 5534.738494] Lustre: server umount lustre-OST0000 complete [ 5534.745228] Lustre: Skipped 1 previous similar message [ 5538.277506] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5544.418267] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5544.434387] LustreError: Skipped 1 previous similar message [ 5547.227658] LustreError: 183143:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776270674 with bad export cookie 4814546055387990208 [ 5547.228908] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5547.236514] LustreError: 183143:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5554.345615] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:31:20 (1776270680) [ 5566.878120] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 5574.070903] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 5582.075754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5587.394621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5591.780701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5596.591296] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5602.469465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5602.518285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5602.714774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5602.739820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5602.817217] Lustre: lustre-MDT0000: new disk, initializing [ 5602.913849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5602.919308] Lustre: Skipped 2 previous similar messages [ 5602.928795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5606.440823] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 5613.936405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5613.987147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5614.060593] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5614.259634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5617.434450] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 5624.473284] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5628.585920] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5633.160423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5633.215727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5633.221792] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5636.570116] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 5642.268409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5642.274371] Lustre: Skipped 1 previous similar message [ 5642.278479] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5644.057142] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5649.892338] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5649.896656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5649.902039] Lustre: Skipped 2 previous similar messages [ 5649.910253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5649.913605] Lustre: Skipped 6 previous similar messages [ 5655.381229] Lustre: server umount lustre-OST0000 complete [ 5655.383795] Lustre: Skipped 2 previous similar messages [ 5667.329486] LustreError: 191707:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776270794 with bad export cookie 4814546055387991286 [ 5667.334748] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5667.343392] LustreError: 191707:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5676.648148] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 5678.619916] Key type lgssc unregistered [ 5678.854342] LNet: 193571:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5678.860371] LNet: Removed LNI 192.168.206.116@tcp [ 5679.389909] Key type .llcrypt unregistered [ 5679.394921] Key type ._llcrypt unregistered [ 5694.271298] alg: No test for adler32 (adler32-zlib) [ 5695.027628] Key type ._llcrypt registered [ 5695.029559] Key type .llcrypt registered [ 5695.095189] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 5695.777363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5696.031938] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 5696.202492] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 5696.205567] LNet: Accept secure, port 988 [ 5697.848231] Key type lgssc registered [ 5698.479575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5703.677619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5704.947730] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5704.988929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5707.759017] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 5710.307432] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5714.121572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5714.479681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5717.842651] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 5723.364624] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5728.007279] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5732.912577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5733.201665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5735.297542] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 5736.897457] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 5742.521606] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5766.113858] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5766.118100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5766.122612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5766.127097] Lustre: Skipped 1 previous similar message [ 5767.497584] Lustre: server umount lustre-OST0000 complete [ 5770.722847] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5770.732916] LustreError: Skipped 1 previous similar message [ 5770.735447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5770.744775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5770.751267] Lustre: Skipped 1 previous similar message [ 5776.295557] Lustre: server umount lustre-MDT0000 complete [ 5776.353814] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5776.365213] LustreError: Skipped 1 previous similar message [ 5778.989782] LustreError: 195012:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776270906 with bad export cookie 4834132808328124376 [ 5778.991789] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5779.004850] LustreError: 195012:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5779.230637] Lustre: server umount lustre-MDT0001 complete [ 5787.381468] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 5789.276362] Key type lgssc unregistered [ 5789.472579] LNet: 198704:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5789.477834] LNet: Removed LNI 192.168.206.116@tcp [ 5790.079871] Key type .llcrypt unregistered [ 5790.082636] Key type ._llcrypt unregistered [ 5806.171987] alg: No test for adler32 (adler32-zlib) [ 5806.927958] Key type ._llcrypt registered [ 5806.929850] Key type .llcrypt registered [ 5806.986487] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 5807.652711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5807.974769] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 5808.104562] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 5808.107651] LNet: Accept secure, port 988 [ 5809.721233] Key type lgssc registered [ 5810.356470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5814.886526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5816.119598] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5816.160146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5819.129131] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 5821.418851] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5824.964674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5825.246679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5828.300283] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 5833.267457] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5836.872958] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5841.058557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5841.273359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5844.308322] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 5846.383114] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 5849.698113] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5860.834857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5860.836049] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5860.846446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5860.856748] Lustre: Skipped 1 previous similar message [ 5861.687083] Lustre: lustre-OST0000: Not available for connect from 192.168.206.16@tcp (stopping) [ 5864.191392] Lustre: server umount lustre-OST0000 complete [ 5869.198513] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9363462c8800.ost_server_uuid 40 [ 5870.231797] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5881.825065] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5881.828982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5881.836718] Lustre: Skipped 1 previous similar message [ 5881.840138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5887.457725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5887.460918] Lustre: Skipped 3 previous similar messages [ 5887.893157] Lustre: server umount lustre-MDT0000 complete [ 5890.294219] LustreError: 200145:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776271017 with bad export cookie 10192741210331707759 [ 5890.297305] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5890.304585] LustreError: 200145:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5890.524106] Lustre: server umount lustre-MDT0001 complete [ 5896.253518] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 12:37:02 (1776271022) [ 5900.283680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5900.595151] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5900.637341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5903.730060] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 5905.890085] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5910.114849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5910.414668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5913.275491] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 5918.932613] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5922.700408] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5927.195121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5927.395333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5930.313310] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 5930.478117] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 5935.421339] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5939.764252] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5939.823946] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5939.830549] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5939.913232] Lustre: lustre-OST0001: new disk, initializing [ 5939.916409] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5942.887502] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 5945.882458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5945.892320] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5949.069416] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5950.949105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5950.950237] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5950.956942] Lustre: Skipped 1 previous similar message [ 5950.961575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5950.968127] Lustre: Skipped 1 previous similar message [ 5956.788583] Lustre: server umount lustre-OST0000 complete [ 5959.980884] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.206.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5960.676342] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5960.818797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5963.409453] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5963.574909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5970.216953] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.206.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5970.231770] LustreError: Skipped 5 previous similar messages [ 5976.545756] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5976.551351] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5976.564176] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5976.568702] Lustre: Skipped 4 previous similar messages [ 5981.943522] Lustre: server umount lustre-OST0001 complete [ 5986.785686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5986.793202] Lustre: Skipped 2 previous similar messages [ 5990.531177] Lustre: server umount lustre-MDT0000 complete [ 5991.906827] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5991.918932] LustreError: Skipped 7 previous similar messages [ 5992.863746] LustreError: 204140:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776271119 with bad export cookie 10192741210331708823 [ 5992.864263] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5992.877909] LustreError: 204140:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5997.250431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6000.049389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6002.367308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6004.646374] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6008.471807] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 12:38:54 (1776271134) [ 6012.294556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6012.459382] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6012.487648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6012.589075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6012.591397] Lustre: Skipped 1 previous similar message [ 6014.825512] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6019.394124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6019.447897] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6019.476638] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6021.919959] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6025.815619] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6028.653646] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6032.184163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6032.300108] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6034.383813] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6038.138598] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6041.636967] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:129 [ 6041.734367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6041.773897] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6044.367593] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6048.575347] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6051.811129] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6051.817418] LustreError: Skipped 1 previous similar message [ 6051.817908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6051.829391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6051.831771] Lustre: Skipped 1 previous similar message [ 6051.846862] Lustre: Skipped 5 previous similar messages [ 6051.917224] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3 to 0x0:33 [ 6056.113195] Lustre: server umount lustre-OST0000 complete [ 6056.118382] Lustre: Skipped 1 previous similar message [ 6056.933646] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6056.943069] LustreError: Skipped 1 previous similar message [ 6072.291180] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6072.292711] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6072.295560] LustreError: Skipped 1 previous similar message [ 6072.304908] Lustre: Skipped 1 previous similar message [ 6086.156264] LustreError: 211376:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776271213 with bad export cookie 10192741210331710202 [ 6086.161065] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6086.169535] LustreError: 211376:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6089.447165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6091.608683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6093.855682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6096.298470] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6099.966293] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 12:40:26 (1776271226) [ 6108.686488] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 6113.364116] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 6119.230774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6123.128525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6126.706040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6130.459587] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6134.600466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6134.647313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6134.806584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6134.827434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6134.877206] Lustre: lustre-MDT0000: new disk, initializing [ 6134.947034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6134.953129] Lustre: Skipped 3 previous similar messages [ 6134.968921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6137.918516] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6145.160142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6145.211569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6145.259219] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6145.292782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6145.297148] Lustre: Skipped 1 previous similar message [ 6145.348152] Lustre: lustre-MDT0001: new disk, initializing [ 6145.426884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6145.439712] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6148.033859] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6154.296163] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6157.804891] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6161.691446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6161.736358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6161.743311] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6161.927876] Lustre: lustre-OST0000: new disk, initializing [ 6161.931383] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6164.508780] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6169.611435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6169.617730] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6170.745686] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6173.665687] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6173.801297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6176.864134] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6177.001308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6179.811263] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6179.815511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6179.815908] LustreError: Skipped 2 previous similar messages [ 6179.819279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6179.822081] Lustre: Skipped 3 previous similar messages [ 6179.841648] Lustre: Skipped 7 previous similar messages [ 6184.131931] Lustre: server umount lustre-OST0000 complete [ 6184.136121] Lustre: Skipped 3 previous similar messages [ 6194.804809] LustreError: 218792:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776271321 with bad export cookie 10192741210331711896 [ 6194.808120] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6194.811963] LustreError: 218792:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6200.031653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6200.319448] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6200.329340] LustreError: Skipped 8 previous similar messages [ 6200.365847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6200.370939] Lustre: Skipped 2 previous similar messages [ 6203.430037] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6208.668308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6211.291646] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6215.902710] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6219.261436] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6222.849953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6225.359223] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6229.640405] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6231.691760] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6231.798668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6233.908541] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6234.018898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6235.625133] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6235.631930] LustreError: Skipped 1 previous similar message [ 6240.409106] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6240.519331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6242.492346] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6269.168340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6272.785290] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6278.857636] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6280.294160] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6288.819393] Lustre: server umount lustre-OST0000 complete [ 6288.821367] Lustre: Skipped 3 previous similar messages [ 6293.986181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6293.990898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6293.998760] Lustre: Skipped 5 previous similar messages [ 6294.008586] Lustre: Skipped 5 previous similar messages [ 6301.027517] LustreError: 223136:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776271428 with bad export cookie 10192741210331712743 [ 6301.031301] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6301.045475] LustreError: 223136:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6308.729509] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 12:43:54 (1776271434) [ 6313.554911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6316.526744] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6321.624334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6324.422270] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6329.394807] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6333.132510] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6337.228348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6337.411643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6337.415506] Lustre: Skipped 5 previous similar messages [ 6339.916598] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6344.128868] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6346.710990] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6346.836777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6349.509372] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6353.803269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 4 sec [ 6357.456743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6357.509431] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6357.516229] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 6357.607437] Lustre: lustre-OST0001: new disk, initializing [ 6357.610730] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 6360.073646] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6364.697305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6364.709099] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6366.275525] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6368.851653] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 6368.982381] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6371.863314] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 6372.025947] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6374.884483] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6374.889784] LustreError: Skipped 1 previous similar message [ 6380.005807] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6380.015229] LustreError: Skipped 6 previous similar messages [ 6382.756600] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 6382.900808] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6385.318420] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 6385.454442] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6411.347651] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6414.292879] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6416.870683] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6416.885414] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to (at 0@lo) [ 6416.886048] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6419.053450] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6419.927840] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6421.987587] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6421.999146] Lustre: Skipped 3 previous similar messages [ 6422.002734] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6422.010299] Lustre: Skipped 5 previous similar messages [ 6426.997991] Lustre: server umount lustre-OST0001 complete [ 6427.004156] Lustre: Skipped 3 previous similar messages [ 6455.266426] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6467.034380] LustreError: 232917:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776271594 with bad export cookie 10192741210331713702 [ 6467.043170] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6467.049046] LustreError: 232917:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6481.376199] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6485.818314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6488.719835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6491.468548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6493.923699] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6498.468053] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 12:47:04 (1776271624) [ 6502.586302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6502.781119] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6502.817439] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6505.555233] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6511.461307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6511.518879] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6511.542232] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6514.600961] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6521.447676] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6526.280751] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6531.644444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6531.944939] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6536.524627] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6543.194313] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6550.219412] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6550.254152] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6552.699912] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6556.825612] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6557.697635] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6560.443645] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 6560.549829] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6563.008440] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 6563.124382] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6566.838834] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff936345aab800.ost_server_uuid 40 [ 6567.857864] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff936345aab800.ost_server_uuid in FULL state after 0 sec [ 6568.472504] Lustre: Permanently deactivating lustre-OST0001 [ 6568.480541] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 6570.369309] Lustre: Permanently reactivating lustre-OST0001 [ 6574.051347] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6574.059487] LustreError: Skipped 4 previous similar messages [ 6604.256124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6606.929446] LustreError: 238283:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776271734 with bad export cookie 10192741210331714962 [ 6606.929707] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6606.937786] LustreError: 238283:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6610.055858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6611.865613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6613.541796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6615.500910] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6618.906787] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 12:49:04 (1776271744) [ 6619.403201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6622.679756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6622.836476] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6622.853419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6622.858345] Lustre: Skipped 5 previous similar messages [ 6622.962432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6622.966865] Lustre: Skipped 6 previous similar messages [ 6625.265264] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6630.505272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6630.583666] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6633.136089] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6637.056678] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6639.987369] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6643.602710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6643.647380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6643.802466] Lustre: Permanently deactivating lustre-OST0000 [ 6646.368900] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6650.660519] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6652.939285] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6654.762850] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6657.190246] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6659.038379] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:97 [ 6661.163705] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6662.190520] Lustre: Permanently reactivating lustre-OST0000 [ 6666.244028] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6666.247544] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6666.252299] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 6666.254249] Lustre: Skipped 1 previous similar message [ 6666.280136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6666.303102] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 6679.523135] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6679.544115] LustreError: Skipped 37 previous similar messages [ 6681.569525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6681.572326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6681.577486] Lustre: Skipped 14 previous similar messages [ 6681.590342] Lustre: Skipped 26 previous similar messages [ 6695.392119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6695.498190] Lustre: server umount lustre-OST0000 complete [ 6695.500937] Lustre: Skipped 8 previous similar messages [ 6706.775905] LustreError: 245123:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776271833 with bad export cookie 10192741210331716565 [ 6706.781851] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6706.785042] LustreError: 245123:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6715.137156] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 6717.457315] Key type lgssc unregistered [ 6717.737596] LNet: 250066:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6717.748902] LNet: Removed LNI 192.168.206.116@tcp [ 6718.394946] Key type .llcrypt unregistered [ 6718.400310] Key type ._llcrypt unregistered [ 6732.757674] alg: No test for adler32 (adler32-zlib) [ 6733.511505] Key type ._llcrypt registered [ 6733.513123] Key type .llcrypt registered [ 6733.567245] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 6734.073981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6734.348248] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 6734.444823] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 6734.446949] LNet: Accept secure, port 988 [ 6736.048229] Key type lgssc registered [ 6736.486074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6737.971634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6741.320766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6742.614055] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6742.683608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6745.247236] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6748.131751] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6750.460823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6750.507428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6750.588383] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6750.604722] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6750.610809] Lustre: Permanently deactivating lustre-MDT0001 [ 6750.814289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6750.909964] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6750.914374] LustreError: 251608:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6750.923799] LustreError: 251608:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6750.931473] LustreError: 251608:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -108 [ 6753.512234] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6759.086691] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6762.893285] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6767.134187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6767.413201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6771.699140] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6774.568769] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6774.576631] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 6776.639272] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6779.874842] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6780.590484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6780.707303] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6781.745198] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:99 to 0x2c0000400:129 [ 6781.748339] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:129 [ 6781.792229] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:131 to 0x280000400:161 [ 6783.199960] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6787.192642] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6789.112815] Lustre: Permanently reactivating lustre-MDT0001 [ 6789.121291] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 6789.125279] Lustre: Skipped 3 previous similar messages [ 6792.448077] Lustre: lustre-MDT0001: Client 9926a03a-4744-4412-8fb7-2c14993d3c0e (at 192.168.206.16@tcp) reconnecting [ 6796.263892] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6797.150697] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6800.119231] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 6800.269430] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6802.851604] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 6802.997338] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6804.472686] Lustre: Permanently deactivating lustre-MDT0001 [ 6804.485346] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 6804.489169] Lustre: Skipped 2 previous similar messages [ 6813.723970] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6822.057630] Lustre: server umount lustre-MDT0000 complete [ 6822.369317] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6822.374397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6822.380686] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6822.884253] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6824.204662] LustreError: 255153:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776271951 with bad export cookie 3503812716910581729 [ 6824.207952] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6824.211257] LustreError: 255153:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6824.393629] Lustre: server umount lustre-MDT0001 complete [ 6826.700446] Lustre: server umount lustre-OST0000 complete [ 6833.061421] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:52:39 (1776271959) [ 6838.535541] Lustre: server umount lustre-OST0001 complete [ 6842.562207] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 6846.933556] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 6852.066700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6856.201414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6859.550603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6863.426985] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6867.520241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6867.562055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6867.696676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6867.701340] Lustre: Skipped 2 previous similar messages [ 6867.719742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6867.766911] Lustre: lustre-MDT0000: new disk, initializing [ 6867.819990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6867.835433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6870.174163] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6876.778039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6876.842399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6876.908578] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6876.951063] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6876.955496] Lustre: Skipped 1 previous similar message [ 6877.003254] Lustre: lustre-MDT0001: new disk, initializing [ 6877.074621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6877.100067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6877.110607] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6879.470257] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6884.985994] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6887.892701] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6891.356549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6891.398462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6891.403218] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6891.549567] Lustre: lustre-OST0000: new disk, initializing [ 6891.553462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6893.458765] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6894.353356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6894.359693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6898.609249] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6899.752097] LustreError: 260083:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6905.006283] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6905.043983] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6905.048371] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 6905.119485] Lustre: lustre-OST0001: new disk, initializing [ 6905.121971] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 6905.157049] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6905.159928] Lustre: Skipped 1 previous similar message [ 6907.043030] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 6909.848050] LustreError: 260083:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 6909.868216] LustreError: 260083:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6912.457989] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6913.284664] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6914.587865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6914.601759] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6919.968180] LustreError: 260083:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 6924.100068] Lustre: lustre-OST0001: Not available for connect from 192.168.206.16@tcp (stopping) [ 6924.772379] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6924.777227] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6924.780314] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6924.785510] Lustre: Skipped 3 previous similar messages [ 6924.789844] Lustre: Skipped 1 previous similar message [ 6927.026526] Lustre: server umount lustre-OST0001 complete [ 6929.899053] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6929.907900] LustreError: Skipped 4 previous similar messages [ 6940.132563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6940.136489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6940.138980] Lustre: Skipped 1 previous similar message [ 6945.251840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6945.255584] Lustre: Skipped 2 previous similar messages [ 6946.153247] Lustre: server umount lustre-OST0000 complete [ 6948.833465] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6948.840410] LustreError: Skipped 1 previous similar message [ 6948.842677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6950.369909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6950.375476] Lustre: Skipped 2 previous similar messages [ 6960.608836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6960.611312] Lustre: Skipped 3 previous similar messages [ 6962.144118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6962.192068] Lustre: server umount lustre-MDT0000 complete [ 6963.567173] LustreError: 260067:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776272090 with bad export cookie 3503812716910584494 [ 6963.570391] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6963.573852] LustreError: 260067:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6975.047420] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 6976.150838] Key type lgssc unregistered [ 6976.270424] LNet: 265271:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6976.273691] LNet: Removed LNI 192.168.206.116@tcp [ 6976.578942] Key type .llcrypt unregistered [ 6976.580734] Key type ._llcrypt unregistered [ 6990.147600] alg: No test for adler32 (adler32-zlib) [ 6990.900326] Key type ._llcrypt registered [ 6990.901718] Key type .llcrypt registered [ 6990.947458] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 6997.684346] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 6998.143717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6998.312360] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 6998.412896] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 6998.415139] LNet: Accept secure, port 988 [ 7000.008112] Key type lgssc registered [ 7000.430620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7004.181364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7007.081812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7009.398326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7011.897153] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7014.499181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7014.523178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7015.611453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7015.628302] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7015.671104] Lustre: lustre-MDT0000: new disk, initializing [ 7015.709993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7015.718444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7017.314215] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7021.997939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7022.023701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7022.062449] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7022.085519] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7022.088375] Lustre: Skipped 1 previous similar message [ 7022.126690] Lustre: lustre-MDT0001: new disk, initializing [ 7022.161897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7022.175181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7022.180409] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7023.730762] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7027.863502] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7030.174848] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7032.851233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7032.886667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7032.890388] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7032.988962] Lustre: lustre-OST0000: new disk, initializing [ 7032.991781] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7033.022140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7034.627317] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7038.887526] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7040.656515] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7041.025137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7041.029509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7041.766405] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 7043.344675] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7043.431065] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7046.113196] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7046.118506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7046.124120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7046.126187] Lustre: Skipped 1 previous similar message [ 7050.332773] Lustre: server umount lustre-OST0000 complete [ 7052.768804] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7052.772058] LustreError: Skipped 1 previous similar message [ 7052.774068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7052.778788] Lustre: Skipped 1 previous similar message [ 7052.781584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7056.353093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7056.353563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7061.473561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7061.476535] Lustre: Skipped 2 previous similar messages [ 7066.080152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7066.122285] Lustre: server umount lustre-MDT0000 complete [ 7067.576238] LustreError: 269351:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776272194 with bad export cookie 14051983322450692822 [ 7067.579129] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7067.581737] LustreError: 269351:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7067.710682] Lustre: server umount lustre-MDT0001 complete [ 7070.785777] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 12:56:37 (1776272197) [ 7073.181613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7073.340953] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7073.362572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7074.968146] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7078.085136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7078.224859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7079.734680] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7082.431263] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7084.370404] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7086.707715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7086.835277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7088.339479] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7091.132931] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7103.457978] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7103.458313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7103.461898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7103.467536] Lustre: Skipped 1 previous similar message [ 7103.470391] Lustre: Skipped 2 previous similar messages [ 7107.613517] Lustre: server umount lustre-OST0000 complete [ 7109.303127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7114.065043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7114.197273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7115.829273] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7119.039279] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7119.340511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7119.346524] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7119.347199] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 7119.352398] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 7119.360150] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 7119.364832] Lustre: Skipped 1 previous similar message [ 7124.449871] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7124.452291] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7124.453252] LustreError: Skipped 1 previous similar message [ 7124.458429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7124.461572] Lustre: Skipped 2 previous similar messages [ 7128.665698] Lustre: server umount lustre-OST0000 complete [ 7134.689820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7134.689959] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7134.695173] Lustre: Skipped 1 previous similar message [ 7137.481170] LustreError: 273896:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776272264 with bad export cookie 14051983322450693662 [ 7137.484502] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7137.486644] LustreError: 273896:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7137.611334] Lustre: server umount lustre-MDT0001 complete [ 7137.612987] Lustre: Skipped 1 previous similar message [ 7142.377209] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 7143.491178] Key type lgssc unregistered [ 7143.611532] LNet: 279325:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7143.615974] LNet: Removed LNI 192.168.206.116@tcp [ 7143.950055] Key type .llcrypt unregistered [ 7143.951330] Key type ._llcrypt unregistered [ 7153.577092] alg: No test for adler32 (adler32-zlib) [ 7154.328452] Key type ._llcrypt registered [ 7154.329816] Key type .llcrypt registered [ 7154.374063] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 7154.848752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7155.025535] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 7155.112054] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 7155.113823] LNet: Accept secure, port 988 [ 7156.696089] Key type lgssc registered [ 7157.081925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7159.876700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7161.041877] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7161.062646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7162.633654] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7165.696770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7165.809313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7167.335078] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7169.974242] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7172.054189] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7174.447102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7174.555918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7175.932574] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7176.613935] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:97 [ 7178.313874] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7191.009699] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7191.011252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7191.013279] LustreError: Skipped 1 previous similar message [ 7191.017540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7191.020827] Lustre: Skipped 1 previous similar message [ 7196.131680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7196.133765] Lustre: Skipped 1 previous similar message [ 7196.515634] Lustre: server umount lustre-OST0000 complete [ 7201.249829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7201.250660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7201.254803] Lustre: Skipped 2 previous similar messages [ 7203.910416] Lustre: server umount lustre-MDT0000 complete [ 7205.109841] LustreError: 280766:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776272332 with bad export cookie 3383062154934467807 [ 7205.112323] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7205.116191] LustreError: 280766:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7205.243853] Lustre: server umount lustre-MDT0001 complete [ 7209.882534] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 7210.901369] Key type lgssc unregistered [ 7211.015277] LNet: 285209:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7211.018366] LNet: Removed LNI 192.168.206.116@tcp [ 7211.349130] Key type .llcrypt unregistered [ 7211.350744] Key type ._llcrypt unregistered [ 7217.746187] alg: No test for adler32 (adler32-zlib) [ 7218.497262] Key type ._llcrypt registered [ 7218.498276] Key type .llcrypt registered [ 7218.540415] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 7219.012176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7219.201793] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 7219.279076] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 7219.280614] LNet: Accept secure, port 988 [ 7220.864140] Key type lgssc registered [ 7221.195452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7223.650538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7224.787557] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7224.801467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7226.225433] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7228.882258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7228.994221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7230.362388] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7232.775282] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7234.732783] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7236.960550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7237.095350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7238.461156] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7240.166717] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 7240.949898] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7249.378623] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7249.378676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7249.383410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7249.386723] Lustre: Skipped 1 previous similar message [ 7249.390515] Lustre: Skipped 1 previous similar message [ 7254.499393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7254.501968] Lustre: Skipped 1 previous similar message [ 7255.141671] Lustre: server umount lustre-OST0000 complete [ 7259.617879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7259.618456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7259.618474] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7259.623291] Lustre: Skipped 1 previous similar message [ 7262.538750] Lustre: server umount lustre-MDT0000 complete [ 7263.939794] LustreError: 286454:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776272391 with bad export cookie 5087727266154539105 [ 7263.942168] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7263.948594] LustreError: 286454:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7264.087268] Lustre: server umount lustre-MDT0001 complete [ 7269.381719] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 7270.462702] Key type lgssc unregistered [ 7270.587453] LNet: 290296:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7270.590822] LNet: Removed LNI 192.168.206.116@tcp [ 7270.921128] Key type .llcrypt unregistered [ 7270.922649] Key type ._llcrypt unregistered [ 7277.930123] alg: No test for adler32 (adler32-zlib) [ 7278.681337] Key type ._llcrypt registered [ 7278.682228] Key type .llcrypt registered [ 7278.718466] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 7279.011154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7279.256713] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 7279.335648] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 7279.337644] LNet: Accept secure, port 988 [ 7280.920106] Key type lgssc registered [ 7281.281637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7284.054259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7285.204658] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7285.220464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7286.620326] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7289.546639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7289.684654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7291.229418] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7293.788318] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7295.915965] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7298.217184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7298.328863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7299.820458] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7300.393342] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 7302.471033] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7310.816898] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7310.820320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7310.825159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7314.913385] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7314.914721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7314.917203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7316.383975] Lustre: server umount lustre-OST0000 complete [ 7320.033418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7320.034396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7320.038554] Lustre: Skipped 1 previous similar message [ 7320.042148] Lustre: Skipped 1 previous similar message [ 7325.153052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7325.156658] Lustre: Skipped 1 previous similar message [ 7330.272917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7330.274836] Lustre: Skipped 1 previous similar message [ 7332.320109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7332.361543] Lustre: server umount lustre-MDT0000 complete [ 7333.628987] LustreError: 291540:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776272460 with bad export cookie 16536713025822139496 [ 7333.631538] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7333.633326] LustreError: 291540:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7333.765285] Lustre: server umount lustre-MDT0001 complete [ 7338.573923] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 7339.528689] Key type lgssc unregistered [ 7339.640325] LNet: 295227:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7339.643553] LNet: Removed LNI 192.168.206.116@tcp [ 7339.924379] Key type .llcrypt unregistered [ 7339.926125] Key type ._llcrypt unregistered [ 7348.728225] alg: No test for adler32 (adler32-zlib) [ 7349.482483] Key type ._llcrypt registered [ 7349.484432] Key type .llcrypt registered [ 7349.520082] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 7349.932351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7350.065345] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 7350.138877] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 7350.140530] LNet: Accept secure, port 988 [ 7351.720162] Key type lgssc registered [ 7352.068657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7354.492620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7355.631664] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7355.648284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7357.142373] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7360.001724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7360.105330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7361.480577] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7363.858812] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7365.683698] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7367.967640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7368.086332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7369.561300] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7371.175292] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 7372.224431] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7385.057596] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7385.060549] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7385.065041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7385.573337] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7385.573964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7385.583418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7390.689068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7390.691867] Lustre: Skipped 1 previous similar message [ 7390.933460] Lustre: server umount lustre-OST0000 complete [ 7395.809581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7395.809597] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7395.810067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7395.814301] Lustre: Skipped 1 previous similar message [ 7398.277968] Lustre: server umount lustre-MDT0000 complete [ 7399.471036] LustreError: 296668:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776272526 with bad export cookie 3017276113719438902 [ 7399.473446] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7399.479486] LustreError: 296668:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7399.592786] Lustre: server umount lustre-MDT0001 complete [ 7404.234339] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 7405.211518] Key type lgssc unregistered [ 7405.331278] LNet: 301160:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7405.334613] LNet: Removed LNI 192.168.206.116@tcp [ 7405.625634] Key type .llcrypt unregistered [ 7405.627347] Key type ._llcrypt unregistered [ 7412.080363] alg: No test for adler32 (adler32-zlib) [ 7412.832311] Key type ._llcrypt registered [ 7412.834301] Key type .llcrypt registered [ 7412.873783] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 7413.194341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7413.416102] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 7413.496187] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 7413.498505] LNet: Accept secure, port 988 [ 7415.080108] Key type lgssc registered [ 7415.451085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7418.119396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7419.254976] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7419.268355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7420.603661] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7423.426740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7423.554934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7425.004582] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7427.549370] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7429.267697] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7431.337021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7431.441544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7432.679507] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7434.535511] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 7435.067867] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7443.938155] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7443.938493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7443.941387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7443.944630] Lustre: Skipped 1 previous similar message [ 7443.948417] Lustre: Skipped 1 previous similar message [ 7449.057854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7449.061917] Lustre: Skipped 1 previous similar message [ 7449.683127] Lustre: server umount lustre-OST0000 complete [ 7454.178102] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7454.178366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7454.179094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7454.186143] Lustre: Skipped 1 previous similar message [ 7456.960531] Lustre: server umount lustre-MDT0000 complete [ 7458.178793] LustreError: 302408:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776272585 with bad export cookie 18442565940219624576 [ 7458.181536] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7458.182775] LustreError: 302408:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7458.300455] Lustre: server umount lustre-MDT0001 complete [ 7462.771546] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 7463.686866] Key type lgssc unregistered [ 7463.791247] LNet: 306261:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7463.793330] LNet: Removed LNI 192.168.206.116@tcp [ 7464.052741] Key type .llcrypt unregistered [ 7464.053815] Key type ._llcrypt unregistered [ 7470.389514] alg: No test for adler32 (adler32-zlib) [ 7471.140366] Key type ._llcrypt registered [ 7471.141632] Key type .llcrypt registered [ 7471.174435] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 7471.489465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7471.699677] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 7471.776269] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 7471.777803] LNet: Accept secure, port 988 [ 7473.352176] Key type lgssc registered [ 7473.736819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7476.530517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7477.677205] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7477.691962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7479.172793] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7482.098815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7482.224665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7483.701819] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7486.316267] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7488.215377] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7490.533244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7490.638697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7492.057514] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7493.733463] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 7494.452635] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7503.331425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7503.331994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7503.335948] Lustre: Skipped 1 previous similar message [ 7503.339940] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7508.194267] Lustre: server umount lustre-OST0000 complete [ 7513.056863] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7513.059432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7513.063776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7513.065815] Lustre: Skipped 1 previous similar message [ 7515.522409] Lustre: server umount lustre-MDT0000 complete [ 7516.704970] LustreError: 309476:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776272643 with bad export cookie 4573926360294704681 [ 7516.707597] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7516.710121] LustreError: 309476:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7516.821237] Lustre: server umount lustre-MDT0001 complete [ 7521.160563] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 7522.107992] Key type lgssc unregistered [ 7522.218342] LNet: 311177:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7522.221482] LNet: Removed LNI 192.168.206.116@tcp [ 7522.483325] Key type .llcrypt unregistered [ 7522.484770] Key type ._llcrypt unregistered [ 7526.665232] alg: No test for adler32 (adler32-zlib) [ 7527.416370] Key type ._llcrypt registered [ 7527.417345] Key type .llcrypt registered [ 7527.450167] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7532.723178] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 7537.566413] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 7538.098759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7538.227067] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 7538.307449] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 7538.309347] LNet: Accept secure, port 988 [ 7539.888098] Key type lgssc registered [ 7540.187645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7542.914882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7544.981278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7546.841325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7548.736234] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7550.889468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7550.911356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7551.986077] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7551.994609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7552.026868] Lustre: lustre-MDT0000: new disk, initializing [ 7552.048969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7552.055507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7553.330680] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7557.155165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7557.175749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7557.205661] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7557.221884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7557.224182] Lustre: Skipped 1 previous similar message [ 7557.257796] Lustre: lustre-MDT0001: new disk, initializing [ 7557.287268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7557.299554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7557.304857] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7558.624907] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7562.239906] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7563.947192] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7565.992314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7566.016570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7566.019935] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7566.111208] Lustre: lustre-OST0000: new disk, initializing [ 7566.113086] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7566.139781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7567.521746] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7571.313919] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7572.684703] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7576.052763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7576.055517] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7576.825146] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 7578.169758] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7578.242707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7581.154751] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7581.155282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7581.157559] LustreError: Skipped 1 previous similar message [ 7581.161161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7581.164970] Lustre: Skipped 1 previous similar message [ 7584.984546] Lustre: server umount lustre-OST0000 complete [ 7586.272952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7586.273454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7586.277669] Lustre: Skipped 2 previous similar messages [ 7591.392842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7591.394756] Lustre: Skipped 2 previous similar messages [ 7592.320536] Lustre: server umount lustre-MDT0000 complete [ 7593.441811] LustreError: 314912:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776272720 with bad export cookie 9889820726319345988 [ 7593.444291] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7593.448153] LustreError: 314912:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7593.549177] Lustre: server umount lustre-MDT0001 complete [ 7596.101150] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 13:05:22 (1776272722) [ 7598.164750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7598.302448] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7598.320742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7599.601763] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7602.354182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7602.450798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7603.778706] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7606.146932] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7607.840764] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7609.924621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7611.270717] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7613.458393] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7622.625482] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7622.626648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7622.630957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7622.634600] Lustre: Skipped 1 previous similar message [ 7622.638568] Lustre: Skipped 1 previous similar message [ 7627.090182] Lustre: server umount lustre-OST0000 complete [ 7628.256710] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7628.259019] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7628.262222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7632.865590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7634.436607] Lustre: server umount lustre-MDT0000 complete [ 7635.658774] LustreError: 321446:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776272762 with bad export cookie 9889820726319346828 [ 7635.661287] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7635.662702] LustreError: 321446:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7640.382186] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 7641.299132] Key type lgssc unregistered [ 7641.403296] LNet: 323148:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7641.405986] LNet: Removed LNI 192.168.206.116@tcp [ 7641.685434] Key type .llcrypt unregistered [ 7641.686484] Key type ._llcrypt unregistered [ 7650.748319] alg: No test for adler32 (adler32-zlib) [ 7651.500478] Key type ._llcrypt registered [ 7651.501537] Key type .llcrypt registered [ 7651.540842] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 7656.608492] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 7656.907476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7657.102701] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 7657.164822] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 7657.166179] LNet: Accept secure, port 988 [ 7658.736066] Key type lgssc registered [ 7659.045569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7661.767134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7663.767734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7665.691431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7667.709397] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7669.307490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7670.960947] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7673.180859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7673.207913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7674.292264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7674.302595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7674.340115] Lustre: lustre-MDT0000: new disk, initializing [ 7674.364490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7674.372478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7675.792360] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7679.837060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7679.862390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7679.895092] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7679.911984] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7679.913643] Lustre: Skipped 1 previous similar message [ 7679.939533] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7679.952563] Lustre: lustre-MDT0001: new disk, initializing [ 7679.981195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7679.994069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7679.997173] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7681.346658] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7685.170350] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7686.989088] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7689.085778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7689.111794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7689.114904] Lustre: lustre-OST2710-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7689.217464] Lustre: lustre-OST2710: new disk, initializing [ 7689.219635] Lustre: srv-lustre-OST2710: No data found on store. Initialize space: rc = -61 [ 7689.245408] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7690.563352] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7693.811602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7693.815483] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7694.207549] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 7696.385820] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7696.407173] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7696.409652] Lustre: lustre-OST03e8-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 7696.456289] Lustre: lustre-OST03e8: new disk, initializing [ 7696.458225] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space: rc = -61 [ 7696.486859] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7697.896547] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7700.447670] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 7700.987912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7700.992508] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7703.265496] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [ 7703.340547] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7704.834513] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 40 [ 7704.904508] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7706.332755] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [ 7706.414994] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7707.911633] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 40 [ 7707.992237] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7710.688836] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7710.690918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7710.694410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7711.203083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7711.207855] Lustre: Skipped 1 previous similar message [ 7715.084691] Lustre: server umount lustre-MDT0000 complete [ 7716.323249] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7716.327080] LustreError: Skipped 3 previous similar messages [ 7716.355381] LustreError: 327078:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776272843 with bad export cookie 9935037841239994183 [ 7716.358878] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7716.359561] LustreError: 327078:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7716.492707] Lustre: server umount lustre-MDT0001 complete [ 7722.976129] Lustre: 332570:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776272844/real 1776272844] req@00000000c7767459 x1862557010778624/t0(0) o39->lustre-MDT0001-lwp-OST2710@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776272850 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 7723.000150] Lustre: server umount lustre-OST2710 complete [ 7730.656122] Lustre: 332771:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776272851/real 1776272851] req@0000000038461ca9 x1862557010779008/t0(0) o39->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776272857 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 7730.743414] Lustre: server umount lustre-OST03e8 complete [ 7736.667664] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 7739.584970] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 7742.876428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7745.130107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7747.042310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7749.147148] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7751.421680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7751.441298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7751.522317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7751.533134] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7751.573658] Lustre: lustre-MDT0000: new disk, initializing [ 7751.600757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7751.607631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7752.986877] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7757.042736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7757.066713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7757.096655] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7757.180801] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7758.495360] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7762.204926] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7764.038642] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7766.094965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7766.115440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7766.117968] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7766.196221] Lustre: lustre-OST0000: new disk, initializing [ 7766.197479] Lustre: Skipped 1 previous similar message [ 7766.198844] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7766.200694] Lustre: Skipped 2 previous similar messages [ 7766.220947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7766.224131] Lustre: Skipped 1 previous similar message [ 7767.574797] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7771.144692] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7772.454883] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7774.196382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7774.199349] Lustre: Skipped 1 previous similar message [ 7774.201332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7774.562774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 7775.945860] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7776.013686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7779.297796] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7779.298070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7779.300687] LustreError: Skipped 1 previous similar message [ 7779.304334] Lustre: Skipped 1 previous similar message [ 7779.306658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7779.310209] Lustre: Skipped 5 previous similar messages [ 7782.739069] Lustre: server umount lustre-OST0000 complete [ 7784.417154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7784.417731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7784.421578] Lustre: Skipped 1 previous similar message [ 7784.425307] Lustre: Skipped 1 previous similar message [ 7789.536954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7789.538851] Lustre: Skipped 1 previous similar message [ 7798.240162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7798.280282] Lustre: server umount lustre-MDT0000 complete [ 7799.550070] LustreError: 335670:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776272926 with bad export cookie 9935037841239995800 [ 7799.552346] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7799.553516] LustreError: 335670:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7802.418012] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 13:08:48 (1776272928) [ 7802.957946] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 7803.545972] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 13:08:49 (1776272929) [ 7804.558135] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 7807.134879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7808.450150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7809.730022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7810.971785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7811.345568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7813.482827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7813.568833] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7813.577846] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7813.632676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7814.983354] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7817.604248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7817.633151] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7819.072756] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7821.393893] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7823.232980] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7825.262805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7825.286034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7825.343951] Lustre: Denying initial registration attempt from nid 192.168.206.116@tcp, specified as failover [ 7825.347181] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 7825.349742] LustreError: 343703:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -99 [ 7825.351726] LustreError: 343703:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 7825.353411] LustreError: 343703:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 7825.394251] Lustre: server umount lustre-OST0000 complete [ 7825.396435] Lustre: Skipped 1 previous similar message [ 7825.398461] LustreError: 343703:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 7827.425168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7827.425823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7827.428450] Lustre: Skipped 1 previous similar message [ 7827.433551] Lustre: Skipped 5 previous similar messages [ 7841.248050] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7842.489842] LustreError: 341743:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776272969 with bad export cookie 9935037841239996640 [ 7842.492663] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7842.493087] LustreError: 341743:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7847.336555] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 7848.334375] Key type lgssc unregistered [ 7848.451317] LNet: 344778:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7848.454685] LNet: Removed LNI 192.168.206.116@tcp [ 7848.774281] Key type .llcrypt unregistered [ 7848.775348] Key type ._llcrypt unregistered [ 7853.371069] alg: No test for adler32 (adler32-zlib) [ 7854.124450] Key type ._llcrypt registered [ 7854.125556] Key type .llcrypt registered [ 7854.159527] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 7854.527525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7854.815460] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 7854.899487] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 7854.901614] LNet: Accept secure, port 988 [ 7856.488116] Key type lgssc registered [ 7856.884180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7859.285377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7860.562621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7861.862487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7863.180160] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7863.529992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7869.316988] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 7872.477261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7873.566890] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7873.577047] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7873.637913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7875.151773] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7878.040190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7878.073312] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7878.083680] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7878.177772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7879.641705] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7882.228938] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7884.214236] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7886.520394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7886.545821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7886.628392] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7886.686127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7888.148490] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7890.684410] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7893.986266] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7893.986765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7893.988915] LustreError: Skipped 1 previous similar message [ 7893.993673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7893.996875] Lustre: Skipped 1 previous similar message [ 7897.516809] Lustre: server umount lustre-OST0000 complete [ 7899.105584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7899.106020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7899.110551] Lustre: Skipped 2 previous similar messages [ 7904.224957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7904.228350] Lustre: Skipped 2 previous similar messages [ 7904.904527] Lustre: server umount lustre-MDT0000 complete [ 7906.214886] LustreError: 347641:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776273033 with bad export cookie 9872546072390503884 [ 7906.218295] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7906.219460] LustreError: 347641:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7906.341392] Lustre: server umount lustre-MDT0001 complete [ 7910.097425] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 7911.144305] Key type lgssc unregistered [ 7911.258367] LNet: 351342:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7911.263157] LNet: Removed LNI 192.168.206.116@tcp [ 7911.573145] Key type .llcrypt unregistered [ 7911.574837] Key type ._llcrypt unregistered [ 7920.638979] alg: No test for adler32 (adler32-zlib) [ 7921.391577] Key type ._llcrypt registered [ 7921.393508] Key type .llcrypt registered [ 7921.430867] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 7921.880771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7922.019091] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 7922.097440] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 7922.100514] LNet: Accept secure, port 988 [ 7923.680119] Key type lgssc registered [ 7924.026612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7926.587497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7927.723274] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7927.738685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7929.185526] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7932.024535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7932.150737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7933.587994] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7935.991243] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7937.944228] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7940.074056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7940.182543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7941.640950] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7944.173766] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7952.232190] LustreError: 355450:0:(ldlm_resource.c:1127:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200001b71:0x1:0x0].0xf14dc48e (000000000bab7bc4) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7952.866622] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7952.867411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7952.868516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7952.875098] Lustre: Skipped 2 previous similar messages [ 7955.753624] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.16@tcp (stopping) [ 7955.755618] Lustre: Skipped 2 previous similar messages [ 7957.986287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7957.989744] Lustre: Skipped 2 previous similar messages [ 7958.346418] Lustre: server umount lustre-MDT0000 complete [ 7959.670268] LustreError: 352782:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776273086 with bad export cookie 3485363480668247399 [ 7959.673124] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7959.676611] LustreError: 352782:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7960.866981] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.206.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7960.875032] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.16@tcp (stopping) [ 7963.105154] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7963.105305] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7965.153306] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7965.153519] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7965.157638] LustreError: Skipped 1 previous similar message [ 7965.161032] Lustre: Skipped 2 previous similar messages [ 7965.901911] Lustre: server umount lustre-MDT0001 complete [ 7968.127872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7970.939660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7976.417340] Lustre: Evicted from MGS (at 192.168.206.116@tcp) after server handle changed from 0x305e7f9e92599567 to 0x305e7f9e9259b13d [ 7976.421740] Lustre: MGC192.168.206.116@tcp: Connection restored to 192.168.206.116@tcp (at 0@lo) [ 7976.525073] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7976.547643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7976.568338] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:193 [ 7977.929196] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7980.817144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7980.937963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7982.288560] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 7984.797424] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7987.399713] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7989.341695] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7989.903494] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7991.265371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7991.266860] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7991.267636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7991.268614] Lustre: Skipped 1 previous similar message [ 7997.209696] Lustre: server umount lustre-OST0000 complete [ 8001.505381] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8001.505783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8004.614038] Lustre: server umount lustre-MDT0000 complete [ 8005.902346] LustreError: 352782:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776273132 with bad export cookie 3485363480668254525 [ 8005.905150] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8005.905845] LustreError: 352782:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8010.745936] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 8011.579263] Key type lgssc unregistered [ 8011.685290] LNet: 359290:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8011.687378] LNet: Removed LNI 192.168.206.116@tcp [ 8011.927073] Key type .llcrypt unregistered [ 8011.927895] Key type ._llcrypt unregistered [ 8020.868190] alg: No test for adler32 (adler32-zlib) [ 8021.621423] Key type ._llcrypt registered [ 8021.622934] Key type .llcrypt registered [ 8021.660918] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 8022.101967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8022.246164] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 8022.325216] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 8022.326953] LNet: Accept secure, port 988 [ 8023.904127] Key type lgssc registered [ 8024.220690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8026.672420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8027.811377] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8027.824416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8029.174686] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8031.941100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8032.057393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8033.383293] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8035.849033] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8037.844972] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8042.465361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8042.465769] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8042.466201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8042.468828] Lustre: Skipped 1 previous similar message [ 8047.584972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8047.586786] Lustre: Skipped 2 previous similar messages [ 8052.704942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8052.706730] Lustre: Skipped 1 previous similar message [ 8053.216082] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8053.254993] Lustre: server umount lustre-MDT0000 complete [ 8054.482350] LustreError: 360731:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776273181 with bad export cookie 4120729624774530647 [ 8054.485704] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8054.486127] LustreError: 360731:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8054.613622] Lustre: server umount lustre-MDT0001 complete [ 8059.385544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8059.481303] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8059.494559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8059.562818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8061.030836] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8063.678131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8063.710221] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8063.721167] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8063.793072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8065.103750] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8067.448030] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8069.439651] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8071.782017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8071.841069] LustreError: 13b-9: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8071.843551] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8071.846179] LustreError: 365441:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 8071.847853] LustreError: 365441:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 8071.849310] LustreError: 365441:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 8071.891571] Lustre: server umount lustre-OST0000 complete [ 8071.892678] LustreError: 365441:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8073.853900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8073.910674] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8073.957223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8075.163852] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8077.130974] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:225 [ 8077.423456] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8079.992887] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8080.017528] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8080.019993] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 8080.040711] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8080.064444] Lustre: lustre-OST0001: new disk, initializing [ 8080.066046] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 8081.418909] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8083.817497] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8084.778708] Lustre: server umount lustre-OST0001 complete [ 8087.021270] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8087.023069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8087.023636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8087.031887] Lustre: Skipped 1 previous similar message [ 8092.131140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8092.131690] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8092.133568] Lustre: Skipped 2 previous similar messages [ 8092.138808] LustreError: Skipped 1 previous similar message [ 8097.251797] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8097.256454] LustreError: Skipped 1 previous similar message [ 8100.320093] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8100.380186] Lustre: server umount lustre-OST0000 complete [ 8102.369540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8102.370199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8102.373738] Lustre: Skipped 1 previous similar message [ 8102.376704] Lustre: Skipped 3 previous similar messages [ 8108.893877] LustreError: 363438:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776273235 with bad export cookie 4120729624774531186 [ 8108.896657] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8108.898603] LustreError: 363438:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8109.012896] Lustre: server umount lustre-MDT0001 complete [ 8109.014319] Lustre: Skipped 1 previous similar message [ 8112.472897] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 8113.449096] Key type lgssc unregistered [ 8113.563341] LNet: 368802:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8113.565801] LNet: Removed LNI 192.168.206.116@tcp [ 8113.877383] Key type .llcrypt unregistered [ 8113.878660] Key type ._llcrypt unregistered [ 8124.999120] alg: No test for adler32 (adler32-zlib) [ 8125.750469] Key type ._llcrypt registered [ 8125.751629] Key type .llcrypt registered [ 8125.792861] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 8131.598972] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 8132.087812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8132.200284] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 8132.279445] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 8132.281995] LNet: Accept secure, port 988 [ 8133.856117] Key type lgssc registered [ 8134.208752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8137.095391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8139.154268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8141.035872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8143.045648] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8145.234959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8145.263401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8146.343758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8146.354060] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8146.395074] Lustre: lustre-MDT0000: new disk, initializing [ 8146.422062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8146.429457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8147.896067] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8152.059057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8152.083663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8152.119476] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8152.137457] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8152.139189] Lustre: Skipped 1 previous similar message [ 8152.179984] Lustre: lustre-MDT0001: new disk, initializing [ 8152.212191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8152.224625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8152.228195] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8153.624754] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8157.402288] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8159.378497] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8161.671761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8161.696479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8161.699094] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8161.787864] Lustre: lustre-OST0000: new disk, initializing [ 8161.789703] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8161.816075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8162.102489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8162.105751] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8163.320459] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8167.151366] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8168.594863] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8168.676826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8170.132452] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8170.209617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8172.514701] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8172.514792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8172.517322] LustreError: Skipped 1 previous similar message [ 8172.521927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8172.524790] Lustre: Skipped 1 previous similar message [ 8177.634369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8177.637057] Lustre: Skipped 1 previous similar message [ 8182.754610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8182.756595] Lustre: Skipped 1 previous similar message [ 8185.312108] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8185.376415] Lustre: server umount lustre-OST0000 complete [ 8187.873571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8187.874091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8187.879040] Lustre: Skipped 2 previous similar messages [ 8192.770537] Lustre: server umount lustre-MDT0000 complete [ 8192.992942] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8192.997677] LustreError: Skipped 1 previous similar message [ 8194.057925] LustreError: 372883:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776273321 with bad export cookie 8195111758559116978 [ 8194.060857] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8194.062912] LustreError: 372883:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8194.172713] Lustre: server umount lustre-MDT0001 complete [ 8196.895734] Lustre: DEBUG MARKER: == conf-sanity test 60: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 13:15:23 (1776273323) [ 8198.877475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8200.885755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8208.366134] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 8211.244575] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 8214.475781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8216.701740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8218.700981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8220.773336] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8223.042560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8223.065800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8223.146365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8223.155838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8223.195133] Lustre: lustre-MDT0000: new disk, initializing [ 8223.220366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8223.227128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8224.716249] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8228.717444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8228.736995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8228.761640] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8228.777460] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8228.780598] Lustre: Skipped 1 previous similar message [ 8228.813118] Lustre: lustre-MDT0001: new disk, initializing [ 8228.838603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8228.847976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8228.850092] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8230.077517] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8233.547553] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8235.314723] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8237.352716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8237.373420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8237.376109] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8237.446423] Lustre: lustre-OST0000: new disk, initializing [ 8237.448159] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8237.469500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8238.701345] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8242.161611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8242.164037] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8242.228614] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8243.533827] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8243.615986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8244.967588] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8245.034366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8247.265697] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8247.266809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8247.268052] LustreError: Skipped 1 previous similar message [ 8247.272183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8247.274434] Lustre: Skipped 2 previous similar messages [ 8251.731970] Lustre: server umount lustre-OST0000 complete [ 8254.432822] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8254.435155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8254.438385] Lustre: Skipped 1 previous similar message [ 8257.504926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8257.507232] Lustre: Skipped 2 previous similar messages [ 8267.232094] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8267.271433] Lustre: server umount lustre-MDT0000 complete [ 8268.464857] LustreError: 380641:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776273395 with bad export cookie 8195111758559117818 [ 8268.467106] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8268.470202] LustreError: 380641:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8271.069820] Lustre: DEBUG MARKER: == conf-sanity test 61: large xattr ====================== 13:16:37 (1776273397) [ 8273.641932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8273.767880] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8273.787147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8274.980330] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8277.517363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8278.884580] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8281.144140] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8282.843051] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8284.841458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8286.156414] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8288.371815] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8296.547260] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61.conf-sanity [ 8297.118415] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61.conf-sanity [ 8297.625598] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61.conf-sanity [ 8298.145213] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61.conf-sanity after remounting MDS [ 8298.792693] Lustre: Failing over lustre-MDT0000 [ 8298.999751] Lustre: server umount lustre-MDT0000 complete [ 8299.001211] Lustre: Skipped 1 previous similar message [ 8299.820902] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.206.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8302.561848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8302.562914] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8302.565933] Lustre: Skipped 3 previous similar messages [ 8302.569886] LustreError: Skipped 1 previous similar message [ 8306.995323] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.206.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8306.999911] LustreError: Skipped 1 previous similar message [ 8309.728114] Lustre: 371116:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776273429/real 1776273429] req@0000000006bcee7a x1862557508870080/t0(0) o400->MGC192.168.206.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776273436 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8309.734344] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8312.054618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8315.874196] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8315.874236] Lustre: Evicted from MGS (at 192.168.206.116@tcp) after server handle changed from 0x71bae2b98f8b0542 to 0x71bae2b98f8b091d [ 8315.879637] LustreError: Skipped 13 previous similar messages [ 8315.883936] Lustre: MGC192.168.206.116@tcp: Connection restored to 192.168.206.116@tcp (at 0@lo) [ 8315.974901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8315.977992] Lustre: Skipped 2 previous similar messages [ 8316.018641] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8317.444864] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8318.243685] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8321.000771] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 8321.006806] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 8321.024406] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 8322.932629] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8323.511270] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8325.297351] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61.conf-sanity [ 8325.951088] Lustre: Failing over lustre-MDT0000 [ 8326.107600] Lustre: server umount lustre-MDT0000 complete [ 8326.115398] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8326.115672] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8326.120847] Lustre: Skipped 1 previous similar message [ 8333.216146] Lustre: 371117:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776273453/real 1776273453] req@00000000e9109e78 x1862557508877440/t0(0) o400->MGC192.168.206.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776273460 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8333.222348] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8333.226379] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8333.232910] LustreError: Skipped 9 previous similar messages [ 8339.229485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8339.426110] Lustre: Evicted from MGS (at 192.168.206.116@tcp) after server handle changed from 0x71bae2b98f8b091d to 0x71bae2b98f8b0c1f [ 8339.431383] Lustre: MGC192.168.206.116@tcp: Connection restored to (at 0@lo) [ 8339.435124] Lustre: Skipped 2 previous similar messages [ 8339.539098] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8340.770874] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8340.886744] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8344.548850] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 8344.556942] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 8344.573905] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 8346.385189] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8346.910163] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8349.665833] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8349.666647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8349.669594] Lustre: Skipped 2 previous similar messages [ 8349.669779] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8349.671622] Lustre: Skipped 6 previous similar messages [ 8356.470304] LustreError: 385276:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776273483 with bad export cookie 8195111758559120415 [ 8356.472943] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8356.473915] LustreError: 385276:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8362.976137] Lustre: 391069:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776273484/real 1776273484] req@0000000053ddfb90 x1862557508886464/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776273490 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8363.044937] Lustre: server umount lustre-OST0000 complete [ 8363.046358] Lustre: Skipped 2 previous similar messages [ 8368.027048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8368.198073] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8368.204614] LustreError: Skipped 8 previous similar messages [ 8369.718227] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8372.569559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8374.127405] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8376.602045] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8378.468612] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8380.571146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8380.672611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8380.674651] Lustre: Skipped 3 previous similar messages [ 8381.983566] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8383.781364] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 8384.341572] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8393.187334] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8393.187344] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8393.191447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8393.194724] Lustre: Skipped 4 previous similar messages [ 8405.585106] LustreError: 391918:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776273532 with bad export cookie 8195111758559121157 [ 8405.588320] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8405.589705] LustreError: 391918:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8410.402585] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 8411.320918] Key type lgssc unregistered [ 8411.426327] LNet: 395547:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8411.429024] LNet: Removed LNI 192.168.206.116@tcp [ 8411.709322] Key type .llcrypt unregistered [ 8411.710687] Key type ._llcrypt unregistered [ 8420.798435] alg: No test for adler32 (adler32-zlib) [ 8421.549348] Key type ._llcrypt registered [ 8421.550388] Key type .llcrypt registered [ 8421.588616] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 8421.985914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8422.120016] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 8422.187192] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 8422.189326] LNet: Accept secure, port 988 [ 8423.760117] Key type lgssc registered [ 8424.074887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8426.604892] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8426.608644] LustreError: 397031:0:(osd_handler.c:8215:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 8426.611377] LustreError: 397031:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 8426.614924] LustreError: 397031:0:(obd_mount.c:200:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 8426.617609] LustreError: 397031:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 8426.620508] LustreError: 397031:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8429.024659] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8429.028372] LustreError: 397428:0:(osd_handler.c:8215:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 8429.033537] LustreError: 397428:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-22) [ 8429.037608] LustreError: 397428:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 8429.041038] LustreError: 397428:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 8429.044911] LustreError: 397428:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8434.973839] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 8435.978246] Key type lgssc unregistered [ 8436.090350] LNet: 398384:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8436.094400] LNet: Removed LNI 192.168.206.116@tcp [ 8436.372709] Key type .llcrypt unregistered [ 8436.373776] Key type ._llcrypt unregistered [ 8442.433121] alg: No test for adler32 (adler32-zlib) [ 8443.185485] Key type ._llcrypt registered [ 8443.186912] Key type .llcrypt registered [ 8443.222488] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 8449.071768] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 8449.506783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8449.691274] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 8449.776812] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 8449.778357] LNet: Accept secure, port 988 [ 8451.360140] Key type lgssc registered [ 8451.739726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8454.660252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8456.910876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8458.966338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8461.088511] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8463.240558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8463.265440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8464.345395] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8464.356377] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8464.391439] Lustre: lustre-MDT0000: new disk, initializing [ 8464.413923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8464.420317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8465.759445] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8469.824598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8469.849532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8469.881399] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8469.901194] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8469.903397] Lustre: Skipped 1 previous similar message [ 8469.941277] Lustre: lustre-MDT0001: new disk, initializing [ 8469.969166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8469.980180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8469.983112] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8471.352289] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8475.075459] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8476.914277] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8479.074009] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8479.102327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8479.104955] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8479.190447] Lustre: lustre-OST0000: new disk, initializing [ 8479.192251] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8479.217682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8480.567640] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8484.300056] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8485.759332] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8486.389450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8486.392369] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8486.861258] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 8488.234122] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8488.308387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8491.490408] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8491.492982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8491.494265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8491.497123] Lustre: Skipped 1 previous similar message [ 8495.145675] Lustre: server umount lustre-OST0000 complete [ 8496.609111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8496.609535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8496.612613] Lustre: Skipped 1 previous similar message [ 8496.614432] Lustre: Skipped 1 previous similar message [ 8501.730058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8501.733139] Lustre: Skipped 2 previous similar messages [ 8502.609595] Lustre: server umount lustre-MDT0000 complete [ 8503.917025] LustreError: 401646:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776273630 with bad export cookie 14609461758247404136 [ 8503.919765] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8503.920508] LustreError: 401646:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8504.034252] Lustre: server umount lustre-MDT0001 complete [ 8507.054621] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 8507.757922] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:20:34 (1776273634) [ 8509.948254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8510.119543] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8510.140844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8511.514638] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8514.327957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8514.442498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8515.869515] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8518.460458] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8520.265595] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8522.405667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8522.513663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8523.870677] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8526.309709] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8528.468799] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8528.494633] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8528.497760] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 8528.545232] Lustre: lustre-OST0001: new disk, initializing [ 8528.547167] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 8529.279670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8529.284719] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8529.974708] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8533.754078] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8539.617205] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8539.617286] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8539.619777] LustreError: Skipped 2 previous similar messages [ 8539.620194] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8539.624298] Lustre: Skipped 1 previous similar message [ 8539.629851] Lustre: Skipped 1 previous similar message [ 8541.678317] Lustre: server umount lustre-OST0001 complete [ 8545.764467] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8545.773108] LustreError: Skipped 1 previous similar message [ 8550.883340] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8550.889648] LustreError: Skipped 1 previous similar message [ 8556.001842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8556.002056] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8556.004153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8556.004158] Lustre: Skipped 3 previous similar messages [ 8556.006354] Lustre: Skipped 1 previous similar message [ 8556.010596] LustreError: Skipped 1 previous similar message [ 8556.012918] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8564.194738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8564.195315] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8564.196521] Lustre: Skipped 4 previous similar messages [ 8564.203745] LustreError: Skipped 3 previous similar messages [ 8568.288112] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8568.346134] Lustre: server umount lustre-OST0000 complete [ 8570.848765] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8570.851353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8577.067074] LustreError: 409153:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776273704 with bad export cookie 14609461758247404976 [ 8577.069359] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8577.072256] LustreError: 409153:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8591.328150] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8591.424367] Lustre: server umount lustre-MDT0001 complete [ 8591.425705] Lustre: Skipped 1 previous similar message [ 8596.186763] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 8597.143766] Key type lgssc unregistered [ 8597.241326] LNet: 411155:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8597.243832] LNet: Removed LNI 192.168.206.116@tcp [ 8597.516259] Key type .llcrypt unregistered [ 8597.517513] Key type ._llcrypt unregistered [ 8608.026743] alg: No test for adler32 (adler32-zlib) [ 8608.778362] Key type ._llcrypt registered [ 8608.779352] Key type .llcrypt registered [ 8608.813385] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 8614.685698] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 8615.022809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8615.238760] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 8615.317412] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 8615.320093] LNet: Accept secure, port 988 [ 8616.904124] Key type lgssc registered [ 8617.232279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8620.086038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8622.525392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8624.541306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8626.461504] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8628.617885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8628.638734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8629.717296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8629.728880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8629.766858] Lustre: lustre-MDT0000: new disk, initializing [ 8629.789907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8629.796247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8631.161480] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8635.164903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8635.187551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8635.221320] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8635.236846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8635.238715] Lustre: Skipped 1 previous similar message [ 8635.275870] Lustre: lustre-MDT0001: new disk, initializing [ 8635.306525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8635.317739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8635.320552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8636.666542] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8640.239572] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8642.062238] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8644.267320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8644.295049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8644.300108] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8644.394917] Lustre: lustre-OST0000: new disk, initializing [ 8644.398947] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8644.430436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8645.741252] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8646.071615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8646.074766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8649.357776] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8650.684244] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8650.754668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8652.040143] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8652.109299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8656.354116] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8656.354403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8656.355133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8656.356350] LustreError: Skipped 1 previous similar message [ 8656.359672] Lustre: Skipped 1 previous similar message [ 8658.842353] Lustre: server umount lustre-OST0000 complete [ 8660.960858] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8660.965272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8660.971313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8660.973137] Lustre: Skipped 1 previous similar message [ 8666.184978] Lustre: server umount lustre-MDT0000 complete [ 8666.592908] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8666.596728] LustreError: Skipped 1 previous similar message [ 8667.393113] LustreError: 415234:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776273794 with bad export cookie 6547903758353597986 [ 8667.396689] LustreError: 415234:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8667.396970] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8667.527599] Lustre: server umount lustre-MDT0001 complete [ 8670.137662] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 13:23:16 (1776273796) [ 8673.110439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8675.864174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8676.002291] Lustre: lustre-MDT0000: new disk, initializing [ 8676.012229] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8676.034107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8677.414199] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8680.177553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8680.289647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8681.644274] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8684.008499] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8685.869249] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8690.657525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8690.657642] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8690.658046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8690.658050] Lustre: Skipped 2 previous similar messages [ 8690.661743] Lustre: Skipped 2 previous similar messages [ 8692.673621] Lustre: server umount lustre-MDT0000 complete [ 8693.857171] LustreError: 420368:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776273820 with bad export cookie 6547903758353598826 [ 8693.859783] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8693.860728] LustreError: 420368:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8696.927716] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:23:43 (1776273823) [ 8699.012982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8699.162757] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8699.183471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8700.533177] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8703.255717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8704.853856] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8707.353868] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8709.311061] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8711.449545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8712.912045] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8715.230995] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8722.842530] Lustre: Permanently deactivating lustre-OST0000 [ 8722.846650] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [ 8732.655308] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8733.589484] LustreError: 426226:0:(mgs_llog.c:1551:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002020 [ 8733.592951] LustreError: 426226:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 8733.595116] LustreError: 426226:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 8734.200294] LustreError: 426274:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 8734.202523] LustreError: 426274:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 8740.886571] Lustre: server umount lustre-OST0000 complete [ 8740.887929] Lustre: Skipped 1 previous similar message [ 8741.764480] LustreError: 426523:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 8741.766807] LustreError: 426523:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 8744.416899] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8744.421661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8744.429155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8744.430965] Lustre: Skipped 1 previous similar message [ 8748.512129] Lustre: 426620:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776273869/real 1776273869] req@00000000a54fe6dd x1862558015328512/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776273875 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8749.809298] LustreError: 422966:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776273876 with bad export cookie 6547903758353599365 [ 8749.812115] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8749.814495] LustreError: 422966:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8755.168133] Lustre: 426822:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776273876/real 1776273876] req@000000000a3d0763 x1862558015329408/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776273882 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8755.312106] Lustre: server umount lustre-MDT0001 complete [ 8755.313895] Lustre: Skipped 1 previous similar message [ 8757.932339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8759.237510] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8760.118793] LustreError: 427731:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -6 [ 8762.152717] Lustre: 428076:0:(mgs_llog.c:1311:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.116@tcp [ 8767.968128] Lustre: 428175:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776273889/real 1776273889] req@00000000360fb4fa x1862558015329728/t0(0) o251->MGC192.168.206.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776273895 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8771.607965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8771.749737] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8771.753704] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8771.756609] Lustre: Skipped 1 previous similar message [ 8771.771771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8771.773534] Lustre: Skipped 2 previous similar messages [ 8773.027684] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8775.672539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8775.763185] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8777.267358] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8779.690121] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8781.494627] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8782.125772] Lustre: Permanently reactivating lustre-OST0000 [ 8782.130424] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 8782.132971] Lustre: Skipped 2 previous similar messages [ 8806.368108] LustreError: 430641:0:(osp_dev.c:684:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 8808.800144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8808.910063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8808.911997] Lustre: Skipped 1 previous similar message [ 8810.173329] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8812.576722] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8821.218884] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8821.219803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8821.221639] LustreError: Skipped 1 previous similar message [ 8821.225261] Lustre: Skipped 1 previous similar message [ 8821.226078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8821.229609] Lustre: Skipped 3 previous similar messages [ 8825.198633] Lustre: server umount lustre-OST0000 complete [ 8825.200014] Lustre: Skipped 1 previous similar message [ 8831.457442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8831.457943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8831.459362] Lustre: Skipped 1 previous similar message [ 8831.464782] Lustre: Skipped 2 previous similar messages [ 8833.840070] LustreError: 429589:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776273960 with bad export cookie 6547903758353600310 [ 8833.843789] LustreError: 429589:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8833.844191] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8838.643502] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 8839.578773] Key type lgssc unregistered [ 8839.679323] LNet: 433526:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8839.682148] LNet: Removed LNI 192.168.206.116@tcp [ 8839.972845] Key type .llcrypt unregistered [ 8839.974420] Key type ._llcrypt unregistered [ 8845.860324] alg: No test for adler32 (adler32-zlib) [ 8846.612351] Key type ._llcrypt registered [ 8846.613371] Key type .llcrypt registered [ 8846.649374] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 8852.423592] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 8852.787640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8853.079438] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 8853.157689] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 8853.160759] LNet: Accept secure, port 988 [ 8854.736068] Key type lgssc registered [ 8855.061402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8857.980150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8860.055560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8861.914725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8863.875094] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8865.993047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8866.014276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8867.091182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8867.100385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8867.135359] Lustre: lustre-MDT0000: new disk, initializing [ 8867.158684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8867.164972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8868.516249] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8872.443862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8872.465054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8872.495348] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8872.511252] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8872.513134] Lustre: Skipped 1 previous similar message [ 8872.547923] Lustre: lustre-MDT0001: new disk, initializing [ 8872.582265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8872.592496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8872.595187] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8873.885138] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8877.437437] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8879.158341] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8881.153546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8881.174448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8881.177169] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8881.266754] Lustre: lustre-OST0000: new disk, initializing [ 8881.269676] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8881.298022] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8882.661208] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8884.980725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8884.983582] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8886.320931] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8887.756269] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8887.829681] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8889.159512] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8889.225236] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8890.337685] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8890.337795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8890.340497] LustreError: Skipped 1 previous similar message [ 8890.341098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8890.344712] Lustre: Skipped 1 previous similar message [ 8891.931829] Lustre: server umount lustre-OST0000 complete [ 8893.408833] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8893.411456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8893.416661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8893.419943] Lustre: Skipped 1 previous similar message [ 8895.457193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8895.457538] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8899.269117] Lustre: server umount lustre-MDT0000 complete [ 8900.486153] LustreError: 436787:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776274027 with bad export cookie 14820461324452541948 [ 8900.489227] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8900.489705] LustreError: 436787:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8900.612297] Lustre: server umount lustre-MDT0001 complete [ 8903.233533] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 8903.765831] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:27:10 (1776274030) [ 8905.981634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8906.128925] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8906.150936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8907.548564] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8910.252182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8910.364174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8911.753176] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8914.114448] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8915.915183] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8918.015893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8919.446848] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8921.818224] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8922.428857] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 8923.877041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 8930.785516] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8930.786938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8930.788193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8930.791139] Lustre: Skipped 1 previous similar message [ 8930.795573] Lustre: Skipped 2 previous similar messages [ 8935.905629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8935.907456] Lustre: Skipped 1 previous similar message [ 8936.413915] Lustre: server umount lustre-OST0000 complete [ 8941.025455] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8941.025525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8941.034589] Lustre: Skipped 1 previous similar message [ 8943.684638] Lustre: server umount lustre-MDT0000 complete [ 8944.838797] LustreError: 441369:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776274071 with bad export cookie 14820461324452542802 [ 8944.840932] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8944.842350] LustreError: 441369:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8949.391301] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 8950.328202] Key type lgssc unregistered [ 8950.435349] LNet: 445234:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8950.437954] LNet: Removed LNI 192.168.206.116@tcp [ 8950.722759] Key type .llcrypt unregistered [ 8950.724102] Key type ._llcrypt unregistered [ 8959.217591] alg: No test for adler32 (adler32-zlib) [ 8959.969372] Key type ._llcrypt registered [ 8959.970390] Key type .llcrypt registered [ 8960.004763] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 8960.091435] Key type .llcrypt unregistered [ 8960.092585] Key type ._llcrypt unregistered [ 8966.615794] alg: No test for adler32 (adler32-zlib) [ 8967.367309] Key type ._llcrypt registered [ 8967.368327] Key type .llcrypt registered [ 8967.403326] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 8967.768778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8967.971545] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 8968.051745] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 8968.053375] LNet: Accept secure, port 988 [ 8969.632077] Key type lgssc registered [ 8969.975864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8972.466495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8973.605397] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8973.619730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8974.929206] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8977.569541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8977.682270] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8977.686091] LustreError: Skipped 1 previous similar message [ 8977.688197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8978.722890] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8978.726809] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 8978.727478] LustreError: Skipped 1 previous similar message [ 8979.015670] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8981.354548] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8983.460121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8983.574267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8984.908711] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 8993.762030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8993.763170] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8993.766422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8993.769705] Lustre: Skipped 1 previous similar message [ 8998.228724] Lustre: server umount lustre-OST0000 complete [ 9002.977997] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 9002.980260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9002.983574] Lustre: Skipped 1 previous similar message [ 9002.985154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9004.000980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9004.001346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9005.572479] Lustre: server umount lustre-MDT0000 complete [ 9006.766955] LustreError: 449818:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776274133 with bad export cookie 1590567089578133429 [ 9006.769729] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9006.770484] LustreError: 449818:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9006.886680] Lustre: server umount lustre-MDT0001 complete [ 9011.412132] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 9012.265163] Key type lgssc unregistered [ 9012.366316] LNet: 450823:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9012.368885] LNet: Removed LNI 192.168.206.116@tcp [ 9012.635343] Key type .llcrypt unregistered [ 9012.636874] Key type ._llcrypt unregistered [ 9021.003372] alg: No test for adler32 (adler32-zlib) [ 9021.754365] Key type ._llcrypt registered [ 9021.755326] Key type .llcrypt registered [ 9021.789692] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 9022.114509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9022.361995] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 9022.437181] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 9022.439116] LNet: Accept secure, port 988 [ 9024.016091] Key type lgssc registered [ 9024.337832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9026.812299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9043.296243] LustreError: 452257:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.206.116@tcp: failed processing log, type 1: rc = -5 [ 9074.016139] LustreError: 452257:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.206.116@tcp: failed processing log, type 4: rc = -110 [ 9104.736173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9106.033420] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 9108.369766] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9110.448894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9127.369885] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9127.391315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9127.413951] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 9128.749215] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 9131.613468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9131.731500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9133.064149] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 9135.378550] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9137.133620] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9142.241578] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9142.246698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9142.253709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9142.752951] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9142.756728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9142.763243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9144.802835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9144.805808] Lustre: Skipped 1 previous similar message [ 9145.445584] Lustre: server umount lustre-OST0000 complete [ 9147.360885] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9147.365147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9147.372066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9149.921579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9152.836460] Lustre: server umount lustre-MDT0000 complete [ 9154.038074] LustreError: 452264:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776274281 with bad export cookie 17845005384511149659 [ 9154.041437] LustreError: 452264:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9154.042376] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9154.183192] Lustre: server umount lustre-MDT0001 complete [ 9158.587575] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 9159.636546] Key type lgssc unregistered [ 9159.749309] LNet: 455950:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9159.751888] LNet: Removed LNI 192.168.206.116@tcp [ 9160.050117] Key type .llcrypt unregistered [ 9160.051177] Key type ._llcrypt unregistered [ 9168.389158] alg: No test for adler32 (adler32-zlib) [ 9169.140323] Key type ._llcrypt registered [ 9169.141919] Key type .llcrypt registered [ 9169.174134] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 9169.484809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9169.695058] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 9169.765436] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 9169.767776] LNet: Accept secure, port 988 [ 9171.344078] Key type lgssc registered [ 9171.642571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9174.166869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9175.297379] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9175.310911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9176.640335] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 9179.367745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9179.478876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9180.737020] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 9182.971604] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9184.697773] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9186.717050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9186.813695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9188.052882] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 9190.258270] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9190.949383] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 9199.074037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9199.074997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9199.077566] Lustre: Skipped 2 previous similar messages [ 9199.079967] Lustre: Skipped 1 previous similar message [ 9201.960955] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.16@tcp (stopping) [ 9201.963596] Lustre: Skipped 1 previous similar message [ 9203.521598] Lustre: server umount lustre-MDT0000 complete [ 9204.188086] LustreError: 457434:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.16@tcp arrived at 1776274331 with bad export cookie 7341832230214026943 [ 9204.191782] LustreError: 457434:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9204.194749] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9204.198039] LustreError: Skipped 2 previous similar messages [ 9204.771799] LustreError: 457434:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776274331 with bad export cookie 7341832230214026299 [ 9204.775035] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9204.778258] LustreError: 457434:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9209.313250] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9209.313328] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9209.313426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9209.314049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9209.319134] LustreError: Skipped 1 previous similar message [ 9210.898401] Lustre: server umount lustre-OST0000 complete [ 9212.949175] Lustre: server umount lustre-MDT0001 complete [ 9217.240852] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 9218.212053] Key type lgssc unregistered [ 9218.315298] LNet: 461212:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9218.317606] LNet: Removed LNI 192.168.206.116@tcp [ 9218.572593] Key type .llcrypt unregistered [ 9218.573835] Key type ._llcrypt unregistered [ 9226.864178] alg: No test for adler32 (adler32-zlib) [ 9227.615307] Key type ._llcrypt registered [ 9227.616315] Key type .llcrypt registered [ 9227.649632] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 9228.052731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9228.176183] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 9228.243349] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 9228.244847] LNet: Accept secure, port 988 [ 9229.816078] Key type lgssc registered [ 9230.117601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9232.481202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9233.617688] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9233.632086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9234.980845] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 9237.679752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9237.808983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9239.222281] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 9241.592124] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9243.368320] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9245.393166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9245.499710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9246.779794] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 9248.612894] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:161 [ 9249.028824] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9253.858391] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9253.862624] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9253.864445] Lustre: Skipped 1 previous similar message [ 9255.202797] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.16@tcp (stopping) [ 9258.977173] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9258.979180] Lustre: Skipped 1 previous similar message [ 9264.097745] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9264.100045] Lustre: Skipped 2 previous similar messages [ 9265.120111] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9265.151039] Lustre: server umount lustre-MDT0001 complete [ 9265.442864] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.206.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9269.217298] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9269.217520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9269.217607] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9269.226396] Lustre: Skipped 1 previous similar message [ 9269.227845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9272.786796] Lustre: server umount lustre-OST0000 complete [ 9280.480156] Lustre: 465724:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776274401/real 1776274401] req@00000000709f9a3c x1862558658079872/t0(0) o251->MGC192.168.206.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776274407 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9280.541036] Lustre: server umount lustre-MDT0000 complete [ 9285.707027] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 9286.586849] Key type lgssc unregistered [ 9286.682301] LNet: 466477:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9286.686475] LNet: Removed LNI 192.168.206.116@tcp [ 9286.974009] Key type .llcrypt unregistered [ 9286.975495] Key type ._llcrypt unregistered [ 9295.749598] alg: No test for adler32 (adler32-zlib) [ 9296.500344] Key type ._llcrypt registered [ 9296.501637] Key type .llcrypt registered [ 9296.534068] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 9301.567121] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 9301.970292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9302.107211] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 9302.178240] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 9302.180035] LNet: Accept secure, port 988 [ 9303.760096] Key type lgssc registered [ 9304.097484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9306.851579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9309.343022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9311.951348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9314.176663] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9317.293331] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 9320.254511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9322.557934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9322.590491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9323.685553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9323.695316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9323.735089] Lustre: lustre-MDT0000: new disk, initializing [ 9323.757585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9323.764479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9325.379486] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 9328.973711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9331.477305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9331.513799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9331.519368] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9331.640712] Lustre: lustre-OST0000: new disk, initializing [ 9331.643203] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9331.645858] Lustre: Skipped 1 previous similar message [ 9331.669135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9332.910723] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 9336.862675] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9340.232416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9342.403569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9342.428503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9342.462658] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9342.488222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9342.529645] Lustre: lustre-MDT0001: new disk, initializing [ 9342.550131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9342.555826] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9342.570714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9344.059418] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 9347.658523] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 9347.738727] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9349.100826] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 9349.177754] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 9352.771060] Lustre: server umount lustre-MDT0001 complete [ 9353.700485] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9353.704371] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9353.704727] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9353.710177] LustreError: Skipped 1 previous similar message [ 9358.817782] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9358.821661] LustreError: Skipped 1 previous similar message [ 9363.937695] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9363.941404] LustreError: Skipped 1 previous similar message [ 9365.984705] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9365.987717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9365.991863] Lustre: Skipped 1 previous similar message [ 9365.993658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9369.057264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9369.057702] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9374.177262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9374.177761] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9379.296120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9379.297260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9379.350719] Lustre: server umount lustre-OST0000 complete [ 9392.096109] Lustre: 474874:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776274513/real 1776274513] req@000000001c49f8eb x1862558735679168/t0(0) o251->MGC192.168.206.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776274519 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9392.155814] Lustre: server umount lustre-MDT0000 complete [ 9397.487926] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 9398.400252] Key type lgssc unregistered [ 9398.509321] LNet: 475629:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9398.511657] LNet: Removed LNI 192.168.206.116@tcp [ 9398.786263] Key type .llcrypt unregistered [ 9398.788209] Key type ._llcrypt unregistered [ 9411.343927] alg: No test for adler32 (adler32-zlib) [ 9412.095361] Key type ._llcrypt registered [ 9412.096326] Key type .llcrypt registered [ 9412.137196] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 9412.230522] Key type .llcrypt unregistered [ 9412.231542] Key type ._llcrypt unregistered [ 9418.287077] alg: No test for adler32 (adler32-zlib) [ 9419.038308] Key type ._llcrypt registered [ 9419.039438] Key type .llcrypt registered [ 9419.079686] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 9419.627723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9419.819248] Lustre: Lustre: Build Version: 2.15.8_1_ga99367a [ 9419.896280] LNet: Added LNI 192.168.206.116@tcp [8/256/0/180] [ 9419.897958] LNet: Accept secure, port 988 [ 9421.480129] Key type lgssc registered [ 9421.823330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9424.321915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9426.346806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9428.161176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9430.234444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9430.254079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9431.354973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9431.367510] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9431.413313] Lustre: lustre-MDT0000: new disk, initializing [ 9431.436726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9431.442987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9432.758965] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 9436.664086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9436.686927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9436.717301] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9436.732676] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9436.734541] Lustre: Skipped 1 previous similar message [ 9436.767860] Lustre: lustre-MDT0001: new disk, initializing [ 9436.801285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9436.813289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9436.818182] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9438.065716] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 9441.683151] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9443.585294] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9445.688785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9445.713804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9445.717134] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9445.821981] Lustre: lustre-OST0000: new disk, initializing [ 9445.823804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9445.849956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9447.170501] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 9451.078798] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9453.575606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9453.582246] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9458.659072] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9458.659805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9458.662515] Lustre: Skipped 2 previous similar messages [ 9458.664141] Lustre: Skipped 1 previous similar message [ 9463.779629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9463.784900] Lustre: Skipped 2 previous similar messages [ 9464.713519] Lustre: server umount lustre-MDT0000 complete [ 9465.987122] LustreError: 479110:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776274593 with bad export cookie 3900387977184147887 [ 9465.989900] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9465.991985] LustreError: 479110:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9466.122419] Lustre: server umount lustre-MDT0001 complete [ 9472.992111] Lustre: 482369:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776274594/real 1776274594] req@00000000bc75c8b5 x1862558858360768/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776274600 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9473.060516] Lustre: server umount lustre-OST0000 complete [ 9476.483585] Lustre: DEBUG MARKER: == conf-sanity test 73: failnode to update from mountdata properly ========================================================== 13:36:42 (1776274602) [ 9476.827583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9479.178525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9479.348735] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9479.368301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9480.708864] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 9484.195713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9484.357121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9486.157647] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 9488.912310] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9490.997799] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9493.453647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9493.481504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9493.563622] Lustre: Found index 0 for lustre-OST0000, updating log [ 9493.643837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9495.633659] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 9495.719443] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5 to 0x0:33 [ 9498.727366] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9501.153140] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9501.155851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9501.161143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9506.222678] Lustre: server umount lustre-OST0000 complete [ 9509.857166] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9509.860906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9509.866677] Lustre: Skipped 1 previous similar message [ 9509.869880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9509.873048] Lustre: Skipped 1 previous similar message [ 9516.512813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9516.514574] Lustre: Skipped 3 previous similar messages [ 9517.258831] Lustre: server umount lustre-MDT0000 complete [ 9518.877535] LustreError: 483173:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776274645 with bad export cookie 3900387977184149210 [ 9518.881348] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9518.883912] LustreError: 483173:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9522.765316] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 13:37:29 (1776274649) [ 9525.083436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9527.069603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9529.514432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9531.505907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9538.239409] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 9541.355650] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 9545.371588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9548.653884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9551.509213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9554.627453] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9557.902467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9557.945352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9558.082179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9558.099695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9558.154029] Lustre: lustre-MDT0000: new disk, initializing [ 9558.183584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9558.193094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9559.787743] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 9564.939235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9564.967495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9564.999723] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9565.016894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9565.019808] Lustre: Skipped 1 previous similar message [ 9565.055686] Lustre: lustre-MDT0001: new disk, initializing [ 9565.103370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9565.107275] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9567.168387] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 9572.126024] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9574.746616] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9577.996311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9578.027465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9578.032959] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9578.184260] Lustre: lustre-OST0000: new disk, initializing [ 9578.188552] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9580.358146] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_default_debug -1 all 8 [ 9585.612628] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9587.198718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9587.202718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9587.617860] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 9587.729421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9589.582671] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 9589.665153] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9592.291228] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9592.291626] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9592.295232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9592.295237] Lustre: Skipped 2 previous similar messages [ 9592.295420] LustreError: Skipped 1 previous similar message [ 9592.301116] Lustre: Skipped 2 previous similar messages [ 9596.653090] Lustre: server umount lustre-OST0000 complete [ 9596.655625] Lustre: Skipped 1 previous similar message [ 9600.993400] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9601.001218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9606.247396] LustreError: 493886:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776274733 with bad export cookie 3900387977184150260 [ 9606.252094] LustreError: 166-1: MGC192.168.206.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9606.253810] LustreError: 493886:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9616.342061] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing set_hostid [ 9619.352182] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing load_modules_local [ 9623.474669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9626.846774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9629.597065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9632.145794] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9632.804545] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9374 sec ========== 13:39:19 (1776274759) [ 9648.378165] Lustre: DEBUG MARKER: oleg616-server.virtnet: executing unload_modules_local [ 9649.307989] Key type lgssc unregistered [ 9649.417291] LNet: 498928:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9649.419938] LNet: Removed LNI 192.168.206.116@tcp [ 9649.697228] Key type .llcrypt unregistered [ 9649.698353] Key type ._llcrypt unregistered