[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 488855854 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002374] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.006723] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008014] pid_max: default: 32768 minimum: 301 [ 0.009168] LSM: Security Framework initializing [ 0.010079] Yama: becoming mindful. [ 0.011052] SELinux: Initializing. [ 0.013048] *** VALIDATE selinux *** [ 0.020802] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027213] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028153] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029293] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031016] *** VALIDATE tmpfs *** [ 0.032564] *** VALIDATE proc *** [ 0.034202] *** VALIDATE cgroup *** [ 0.035017] *** VALIDATE cgroup2 *** [ 0.037009] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038223] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040042] Spectre V2 : User space: Vulnerable [ 0.042009] Speculative Store Bypass: Vulnerable [ 0.046711] debug: unmapping init [mem 0xffffffffb9259000-0xffffffffb9260fff] [ 0.048345] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050056] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051025] ... version: 2 [ 0.052017] ... bit width: 48 [ 0.053018] ... generic registers: 4 [ 0.054016] ... value mask: 0000ffffffffffff [ 0.055018] ... max period: 00007fffffffffff [ 0.056015] ... fixed-purpose events: 3 [ 0.057014] ... event mask: 000000070000000f [ 0.058288] rcu: Hierarchical SRCU implementation. [ 0.060538] smp: Bringing up secondary CPUs ... [ 0.061663] x86: Booting SMP configuration: [ 0.062030] .... node #0, CPUs: #1 #2 #3 [ 0.065433] smp: Brought up 1 node, 4 CPUs [ 0.067012] smpboot: Max logical packages: 1 [ 0.068021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.162245] node 0 deferred pages initialised in 90ms [ 0.165138] devtmpfs: initialized [ 0.167118] x86/mm: Memory block size: 128MB [ 0.169881] gcov: version magic: 0x41383552 [ 0.170577] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.174103] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.177451] pinctrl core: initialized pinctrl subsystem [ 0.179175] [ 0.179842] ************************************************************* [ 0.182010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.185012] ** ** [ 0.187018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.190011] ** ** [ 0.192015] ** This means that this kernel is built to expose internal ** [ 0.194012] ** IOMMU data structures, which may compromise security on ** [ 0.196014] ** your system. ** [ 0.199016] ** ** [ 0.202012] ** If you see this message and you are not debugging the ** [ 0.204012] ** kernel, report this immediately to your vendor! ** [ 0.207020] ** ** [ 0.209013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.211026] ************************************************************* [ 0.214018] NET: Registered protocol family 16 [ 0.216442] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.218073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.222062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.225063] cpuidle: using governor menu [ 0.226781] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.227698] PCI: Using configuration type 1 for base access [ 0.228164] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.235228] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.238068] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.243046] cryptd: max_cpu_qlen set to 1000 [ 0.245257] ACPI: Added _OSI(Module Device) [ 0.248016] ACPI: Added _OSI(Processor Device) [ 0.249010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.250013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.254959] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.262341] ACPI: Interpreter enabled [ 0.264072] ACPI: PM: (supports S0 S3 S4 S5) [ 0.266015] ACPI: Using IOAPIC for interrupt routing [ 0.268121] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.271351] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.281940] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.285046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.287020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.291090] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.295384] acpiphp: Slot [2] registered [ 0.297177] acpiphp: Slot [5] registered [ 0.299107] acpiphp: Slot [6] registered [ 0.301243] acpiphp: Slot [7] registered [ 0.303209] acpiphp: Slot [8] registered [ 0.304137] acpiphp: Slot [9] registered [ 0.306233] acpiphp: Slot [10] registered [ 0.308184] acpiphp: Slot [3] registered [ 0.309159] acpiphp: Slot [4] registered [ 0.311177] acpiphp: Slot [11] registered [ 0.313145] acpiphp: Slot [12] registered [ 0.314140] acpiphp: Slot [13] registered [ 0.316165] acpiphp: Slot [14] registered [ 0.318149] acpiphp: Slot [15] registered [ 0.319147] acpiphp: Slot [16] registered [ 0.321176] acpiphp: Slot [17] registered [ 0.323170] acpiphp: Slot [18] registered [ 0.325146] acpiphp: Slot [19] registered [ 0.326135] acpiphp: Slot [20] registered [ 0.328148] acpiphp: Slot [21] registered [ 0.329170] acpiphp: Slot [22] registered [ 0.331167] acpiphp: Slot [23] registered [ 0.333110] acpiphp: Slot [24] registered [ 0.334126] acpiphp: Slot [25] registered [ 0.336115] acpiphp: Slot [26] registered [ 0.337111] acpiphp: Slot [27] registered [ 0.338124] acpiphp: Slot [28] registered [ 0.340097] acpiphp: Slot [29] registered [ 0.341076] acpiphp: Slot [30] registered [ 0.341959] acpiphp: Slot [31] registered [ 0.343072] PCI host bridge to bus 0000:00 [ 0.345066] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.348042] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.352035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.356036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.361031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.366031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.370307] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.375414] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.380999] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.397783] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.404074] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.408025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.412017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.414017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.416602] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.422448] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.425051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.428898] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.435018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.448017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.453022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.460445] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.467019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.474019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.491021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.501589] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.510018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.515015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.547026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.561173] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.574016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.587084] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.610015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.624178] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.636028] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.645023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.669026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.679673] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.690030] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.701016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.718025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.737260] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.747027] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.754027] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.779022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.792920] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.795385] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.797448] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.801403] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.804217] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.809030] iommu: Default domain type: Passthrough [ 0.811512] SCSI subsystem initialized [ 0.813158] ACPI: bus type USB registered [ 0.815126] usbcore: registered new interface driver usbfs [ 0.817080] usbcore: registered new interface driver hub [ 0.819086] usbcore: registered new device driver usb [ 0.821485] pps_core: LinuxPPS API ver. 1 registered [ 0.823014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.827240] PTP clock support registered [ 0.830072] EDAC MC: Ver: 3.0.0 [ 0.833127] PCI: Using ACPI for IRQ routing [ 0.834956] NetLabel: Initializing [ 0.836015] NetLabel: domain hash size = 128 [ 0.838011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.840081] NetLabel: unlabeled traffic allowed by default [ 0.842120] vgaarb: loaded [ 0.843248] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.845015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.853006] clocksource: Switched to clocksource kvm-clock [ 0.961786] VFS: Disk quotas dquot_6.6.0 [ 0.963554] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.966522] *** VALIDATE ramfs *** [ 0.967957] *** VALIDATE hugetlbfs *** [ 0.969922] pnp: PnP ACPI init [ 0.972611] pnp: PnP ACPI: found 6 devices [ 0.994156] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.998088] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.000885] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.003685] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.006225] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.009100] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.012087] NET: Registered protocol family 2 [ 1.014473] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.019445] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.022840] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.028828] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.032710] TCP: Hash tables configured (established 65536 bind 65536) [ 1.035698] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.039462] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.042497] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.045886] NET: Registered protocol family 1 [ 1.050041] RPC: Registered named UNIX socket transport module. [ 1.052404] RPC: Registered udp transport module. [ 1.054381] RPC: Registered tcp transport module. [ 1.056060] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.058832] NET: Registered protocol family 44 [ 1.060303] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.062687] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.064659] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.066957] PCI: CLS 0 bytes, default 64 [ 1.068810] Unpacking initramfs... [ 2.543396] debug: unmapping init [mem 0xffff99153cc54000-0xffff99153ffbffff] [ 2.547607] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.550117] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.554708] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.053782] Initialise system trusted keyrings [ 3.056043] Key type blacklist registered [ 3.058173] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.068625] zbud: loaded [ 3.072245] *** VALIDATE nfs *** [ 3.073760] *** VALIDATE nfs4 *** [ 3.075695] pstore: using deflate compression [ 3.079560] Platform Keyring initialized [ 3.191698] NET: Registered protocol family 38 [ 3.193395] Key type asymmetric registered [ 3.194566] Asymmetric key parser 'x509' registered [ 3.196608] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.199574] io scheduler mq-deadline registered [ 3.201171] io scheduler kyber registered [ 3.202737] io scheduler bfq registered [ 3.204511] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.207397] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.210177] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.213240] ACPI: Power Button [PWRF] [ 3.219248] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.226750] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.244450] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.253106] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.266269] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.294088] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.323738] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.328204] Non-volatile memory driver v1.3 [ 3.329731] Linux agpgart interface v0.103 [ 3.357255] virtio_blk virtio1: [vda] 149568 512-byte logical blocks (76.6 MB/73.0 MiB) [ 3.359690] vda: detected capacity change from 0 to 76578816 [ 3.379135] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.381673] vdb: detected capacity change from 0 to 1073741824 [ 3.394752] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.397237] vdc: detected capacity change from 0 to 2621440000 [ 3.409898] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.412262] vdd: detected capacity change from 0 to 2621440000 [ 3.431223] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.434639] vde: detected capacity change from 0 to 4294967296 [ 3.449762] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.453163] vdf: detected capacity change from 0 to 4294967296 [ 3.463767] libphy: Fixed MDIO Bus: probed [ 3.469137] usbcore: registered new interface driver usbserial_generic [ 3.472876] usbserial: USB Serial support registered for generic [ 3.475336] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.479945] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.481951] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.484699] mousedev: PS/2 mouse device common for all mice [ 3.487887] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.489702] rtc_cmos 00:05: RTC can wake from S4 [ 3.498843] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.502446] rtc_cmos 00:05: registered as rtc0 [ 3.511417] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.512463] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.514297] intel_pstate: CPU model not supported [ 3.521048] hid: raw HID events driver (C) Jiri Kosina [ 3.523542] usbcore: registered new interface driver usbhid [ 3.525702] usbhid: USB HID core driver [ 3.527376] drop_monitor: Initializing network drop monitor service [ 3.529950] Initializing XFRM netlink socket [ 3.532133] NET: Registered protocol family 10 [ 3.535414] Segment Routing with IPv6 [ 3.536886] NET: Registered protocol family 17 [ 3.538959] mpls_gso: MPLS GSO support [ 3.545839] RAS: Correctable Errors collector initialized. [ 3.548192] AVX version of gcm_enc/dec engaged. [ 3.549935] AES CTR mode by8 optimization enabled [ 3.637923] sched_clock: Marking stable (3637897919, 0)->(4624866298, -986968379) [ 3.642175] registered taskstats version 1 [ 3.644206] Loading compiled-in X.509 certificates [ 3.646452] zswap: loaded using pool lzo/zbud [ 3.674589] Key type big_key registered [ 3.687533] Key type encrypted registered [ 3.689183] ima: No TPM chip found, activating TPM-bypass! [ 3.691264] ima: Allocated hash algorithm: sha1 [ 3.693058] ima: No architecture policies found [ 3.694806] evm: Initialising EVM extended attributes: [ 3.696840] evm: security.selinux [ 3.698069] evm: security.ima [ 3.699180] evm: security.capability [ 3.700484] evm: HMAC attrs: 0x1 [ 3.704855] rtc_cmos 00:05: setting system clock to 2026-08-22 05:33:29 UTC (1787376809) [ 3.715416] debug: unmapping init [mem 0xffffffffba203000-0xffffffffba3fffff] [ 3.718326] debug: unmapping init [mem 0xffffffffb8f82000-0xffffffffb9258fff] [ 3.728172] Write protecting the kernel read-only data: 28672k [ 3.731963] debug: unmapping init [mem 0xffffffffb7603000-0xffffffffb77fffff] [ 3.735021] debug: unmapping init [mem 0xffffffffb7f14000-0xffffffffb7ffffff] [ 3.771519] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.779956] systemd[1]: Detected virtualization kvm. [ 3.781784] systemd[1]: Detected architecture x86-64. [ 3.783566] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.808650] systemd[1]: No hostname configured. [ 3.810183] systemd[1]: Set hostname to . [ 3.812026] random: systemd: uninitialized urandom read (16 bytes read) [ 3.814094] systemd[1]: Initializing machine ID from random generator. [ 3.874427] random: ln: uninitialized urandom read (6 bytes read) [ 3.964434] random: systemd: uninitialized urandom read (16 bytes read) [ 3.967747] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.974594] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.979467] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.645893] device-mapper: uevent: version 1.0.3 [ 4.650222] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.436266] virtio_net virtio0 ens2: renamed from eth0 [ 5.479793] random: fast init done [ 5.507760] scsi host0: ata_piix [ 5.563632] scsi host1: ata_piix [ 5.565435] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.568606] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.949878] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.207704] random: crng init done [ 10.209267] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.785358] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.029654] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.310709] SELinux: Disabled at runtime. [ 12.379277] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.388985] systemd[1]: Detected virtualization kvm. [ 12.391078] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.959643] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.964686] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.970549] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.976303] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.980269] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.992629] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.000573] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 13.126508] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.460111] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.881777] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.940801] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.106318] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.122686] EDAC sbridge: Ver: 1.1.2 [ 16.027761] Key type dns_resolver registered [ 16.375648] NFS: Registering the id_resolver key type [ 16.378202] Key type id_resolver registered [ 16.380167] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg628-server login: [ 42.991309] libcfs: loading out-of-tree module taints kernel. [ 43.016180] Key type ._llcrypt registered [ 43.017649] Key type .llcrypt registered [ 43.064697] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 52.354836] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 53.015914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 53.025133] alg: No test for adler32 (adler32-zlib) [ 54.731403] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 55.660521] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 57.384337] Key type lgssc registered [ 58.502014] hrtimer: interrupt took 3703269 ns [ 59.345778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 79.298966] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 123.440921] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 139.660822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 139.727479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 141.168243] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 141.205950] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 141.397517] Lustre: lustre-MDT0000: new disk, initializing [ 141.614164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 141.654066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 146.992177] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 162.654806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 162.820155] Lustre: 6492:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 162.849623] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 162.853069] Lustre: Skipped 1 previous similar message [ 162.933361] Lustre: lustre-MDT0001: new disk, initializing [ 163.016896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 163.065591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 163.070680] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 167.552552] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.952105] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 181.867270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 182.170986] Lustre: lustre-OST0000: new disk, initializing [ 182.177520] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 182.182606] Lustre: 8430:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 182.251649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.928699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 186.935904] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 187.009097] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 188.823343] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.465955] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 204.622563] Lustre: lustre-OST0001: new disk, initializing [ 204.626817] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 204.633889] Lustre: 9505:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 204.704104] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 210.500657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 210.510907] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 210.588907] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 211.010547] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.716782] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 232.471296] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 240.292949] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing check_logdir /tmp/testlogs/ [ 245.548902] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing yml_node [ 251.243579] Lustre: DEBUG MARKER: Client: 2.17.54.117 [ 254.252480] Lustre: DEBUG MARKER: MDS: 2.17.54.117 [ 257.406483] Lustre: DEBUG MARKER: OSS: 2.17.54.117 [ 259.217594] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 01:37:43 EDT 2026 [ 279.005564] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 281.243873] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 297.952881] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.960913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 297.969166] Lustre: Skipped 3 previous similar messages [ 297.989766] Lustre: Skipped 3 previous similar messages [ 303.072917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 303.802996] Lustre: server umount lustre-MDT0000 complete [ 307.835236] LustreError: 6484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787377113 with bad export cookie 6489173630839585716 [ 307.843018] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 307.854952] LustreError: 6484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 308.192334] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 308.204226] Lustre: Skipped 1 previous similar message [ 308.205103] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 308.205109] Lustre: Skipped 3 previous similar messages [ 312.224201] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 312.236431] Lustre: Skipped 2 previous similar messages [ 314.106933] Lustre: server umount lustre-MDT0001 complete [ 318.199816] Lustre: server umount lustre-OST0000 complete [ 323.317202] Lustre: server umount lustre-OST0001 complete [ 332.304754] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 343.139679] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 383.665060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 383.999483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 384.051457] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 384.159451] Lustre: lustre-MDT0000: new disk, initializing [ 384.225396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 384.242239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 388.994524] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 400.108689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 400.267732] Lustre: 15057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 400.333574] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 400.347443] Lustre: Skipped 1 previous similar message [ 400.508528] Lustre: lustre-MDT0001: new disk, initializing [ 400.681218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 400.715416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 400.741185] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 405.551447] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 415.851732] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 422.605408] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 430.129984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 430.488319] Lustre: lustre-OST0000: new disk, initializing [ 430.492780] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 430.500710] Lustre: 17026:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 430.619495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 431.576472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 431.592296] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 431.827978] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 437.658063] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 449.353193] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 456.335743] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 456.649992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 462.142986] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 462.457818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 467.426456] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 467.448642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.476726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 467.939071] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 471.400875] Lustre: server umount lustre-OST0000 complete [ 477.665900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 477.691437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 477.714217] Lustre: Skipped 1 previous similar message [ 477.725592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 477.739642] Lustre: Skipped 1 previous similar message [ 482.469906] Lustre: server umount lustre-MDT0000 complete [ 483.298511] LustreError: 15068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 483.319253] LustreError: 15068:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 486.777162] LustreError: 15049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787377292 with bad export cookie 6489173630839587256 [ 486.783821] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 486.794644] LustreError: 15049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 487.244061] Lustre: server umount lustre-MDT0001 complete [ 491.999652] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:41:36 (1787377296) [ 495.113584] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 497.317323] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:41:41 (1787377301) [ 504.808177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 505.342713] LustreError: 19435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 505.419033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 510.097958] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 510.438280] LustreError: 19436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.559586] LustreError: 19435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 519.748543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 525.176262] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 533.084350] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 540.064546] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 547.603038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 555.298967] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 564.007485] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 574.954035] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 574.961984] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 574.970974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 574.970983] Lustre: Skipped 2 previous similar messages [ 574.972829] Lustre: Skipped 2 previous similar messages [ 580.108738] LustreError: 21257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 580.146571] LustreError: 21257:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 580.264901] Lustre: server umount lustre-OST0000 complete [ 585.185914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 585.212440] Lustre: Skipped 1 previous similar message [ 590.137190] Lustre: server umount lustre-MDT0000 complete [ 590.306989] LustreError: 19431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.318445] LustreError: 19431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 594.061857] LustreError: 19415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787377399 with bad export cookie 6489173630839588138 [ 594.064924] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 594.073949] LustreError: 19415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 594.398712] Lustre: server umount lustre-MDT0001 complete [ 606.126732] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 609.343177] Key type lgssc unregistered [ 609.626539] LNet: 22969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 609.632350] LNetError: 22969:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 609.647110] LNet: Removed LNI 192.168.206.128@tcp [ 610.692294] Key type .llcrypt unregistered [ 610.696174] Key type ._llcrypt unregistered [ 635.276668] Key type ._llcrypt registered [ 635.287463] Key type .llcrypt registered [ 635.440434] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 637.082685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 637.093670] alg: No test for adler32 (adler32-zlib) [ 638.278907] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 638.635741] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 640.319703] Key type lgssc registered [ 641.528905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 650.259219] loop: module loaded [ 650.319699] loop0: detected capacity change from 0 to 204800000 [ 650.589370] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 650.604838] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 650.877129] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 656.625231] loop0: detected capacity change from 0 to 204800000 [ 656.951893] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 656.975789] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 656.995077] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 662.375409] loop0: detected capacity change from 0 to 204800000 [ 667.506610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 667.526594] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 668.752285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 668.818831] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 668.900345] Lustre: lustre-MDT0000: new disk, initializing [ 668.986665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 669.013436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 673.565747] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 682.163942] loop1: detected capacity change from 0 to 204800000 [ 687.666637] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 688.021936] Lustre: lustre-OST07c6: new disk, initializing [ 688.028378] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 688.036693] Lustre: Skipped 1 previous similar message [ 688.044909] Lustre: 26157:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 688.153583] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 696.980218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 697.008852] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 697.046785] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 697.214434] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 704.051713] Lustre: 25145:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 704.070242] Lustre: 25145:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 704.083161] Lustre: 25145:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 704.095238] Lustre: 25145:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 704.102590] Lustre: 25145:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 704.115338] Lustre: 25145:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 722.787941] Lustre: 26673:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 722.802562] Lustre: 26673:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 722.807911] Lustre: 26673:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 722.812379] Lustre: 26673:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 722.825498] Lustre: 26673:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 722.834884] Lustre: 26673:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 727.659785] Lustre: 26673:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 727.676301] Lustre: 26673:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 727.683596] Lustre: 26673:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 727.689933] Lustre: 26673:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.700764] Lustre: 26673:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 727.707847] Lustre: 26673:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.715140] Lustre: 26673:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 727.722460] Lustre: 26673:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.727575] Lustre: 26673:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 727.734168] Lustre: 26673:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.741121] Lustre: 26673:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 727.749727] Lustre: 26673:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 730.925100] Lustre: 25146:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 730.938587] Lustre: 25146:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 730.950485] Lustre: 25146:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 730.959159] Lustre: 25146:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 730.966261] Lustre: 25146:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 730.973844] Lustre: 25146:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 730.980505] Lustre: 25146:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 730.988221] Lustre: 25146:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 730.996230] Lustre: 25146:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 731.010325] Lustre: 25146:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 731.018484] Lustre: 25146:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 731.024370] Lustre: 25146:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 735.603735] Lustre: 25146:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 735.620188] Lustre: 25146:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 735.630249] Lustre: 25146:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 735.637763] Lustre: 25146:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 735.654447] Lustre: 25146:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 735.662507] Lustre: 25146:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 735.671665] Lustre: 25146:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 735.678396] Lustre: 25146:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 735.687669] Lustre: 25146:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 735.697285] Lustre: 25146:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 735.704540] Lustre: 25146:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 735.711453] Lustre: 25146:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.800087] Lustre: 25144:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 746.811116] Lustre: 25144:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 746.821789] Lustre: 25144:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 746.832380] Lustre: 25144:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 746.837716] Lustre: 25144:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 746.843172] Lustre: 25144:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 746.851129] Lustre: 25144:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 746.861309] Lustre: 25144:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 746.867627] Lustre: 25144:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 746.877163] Lustre: 25144:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 746.885951] Lustre: 25144:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 746.892242] Lustre: 25144:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 799.208453] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 799.226957] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 801.493873] Lustre: server umount lustre-OST07c6 complete [ 811.864498] Lustre: server umount lustre-MDT0000 complete [ 829.408640] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 832.263535] Key type lgssc unregistered [ 832.586110] LNet: 28716:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.593202] LNetError: 28716:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 832.605914] LNet: Removed LNI 192.168.206.128@tcp [ 833.415207] Key type .llcrypt unregistered [ 833.421428] Key type ._llcrypt unregistered [ 851.662145] Key type ._llcrypt registered [ 851.665080] Key type .llcrypt registered [ 851.770185] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 853.034606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 853.057960] alg: No test for adler32 (adler32-zlib) [ 854.106917] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 854.336555] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 856.031180] Key type lgssc registered [ 857.216146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 864.473492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 864.498869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 865.884662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 870.255510] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 876.007645] LustreError: 29987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 876.022177] LustreError: 29987:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 877.688834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 877.963656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 881.947582] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 888.826589] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 893.888562] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 900.480497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 900.827139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 905.853808] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 911.092236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 911.991656] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 926.691945] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 926.706820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 926.710137] Lustre: Skipped 1 previous similar message [ 931.197645] Lustre: server umount lustre-OST0000 complete [ 934.368566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 934.375470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 934.382951] Lustre: Skipped 1 previous similar message [ 934.386705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 936.929420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 936.939457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 940.461280] Lustre: server umount lustre-MDT0000 complete [ 942.051575] LustreError: 29986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 942.086925] LustreError: 29986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 943.671076] LustreError: 29967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787377749 with bad export cookie 10097989440973851068 [ 943.679157] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 943.696772] LustreError: 29967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 943.993371] Lustre: server umount lustre-MDT0001 complete [ 953.309761] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 955.721313] Key type lgssc unregistered [ 956.031584] LNet: 33721:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.040686] LNetError: 33721:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.058120] LNet: Removed LNI 192.168.206.128@tcp [ 956.794767] Key type .llcrypt unregistered [ 956.798501] Key type ._llcrypt unregistered [ 974.408398] Key type ._llcrypt registered [ 974.414670] Key type .llcrypt registered [ 974.532257] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 975.762657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 975.780731] alg: No test for adler32 (adler32-zlib) [ 976.761401] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 976.929247] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 978.609074] Key type lgssc registered [ 979.847720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 987.569508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 987.591374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 989.103889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 992.711695] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 998.376648] LustreError: 34996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 998.404176] LustreError: 34996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1000.257719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1000.534510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1005.300235] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1011.938067] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1016.742249] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1022.647854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1023.015554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1028.390223] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1033.281458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1035.565424] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1048.548667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1048.554087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1048.570601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1048.579637] Lustre: Skipped 1 previous similar message [ 1052.963217] Lustre: server umount lustre-OST0000 complete [ 1056.743485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1056.751618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1056.788158] Lustre: Skipped 1 previous similar message [ 1056.803469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1058.785369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1058.790030] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1062.234549] LustreError: 37777:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000fae56a24) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 1063.907070] LustreError: 34990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1063.929527] LustreError: 34990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1064.338761] Lustre: server umount lustre-MDT0000 complete [ 1067.937034] LustreError: 34977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787377873 with bad export cookie 18278977814911109863 [ 1067.941341] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1067.946982] LustreError: 34977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1068.159369] Lustre: server umount lustre-MDT0001 complete [ 1076.866862] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 1079.278862] Key type lgssc unregistered [ 1079.535365] LNet: 38529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1079.541350] LNetError: 38529:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1079.559843] LNet: Removed LNI 192.168.206.128@tcp [ 1080.252312] Key type .llcrypt unregistered [ 1080.254068] Key type ._llcrypt unregistered [ 1100.514753] Key type ._llcrypt registered [ 1100.516664] Key type .llcrypt registered [ 1100.646603] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 1101.410869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1101.490828] alg: No test for adler32 (adler32-zlib) [ 1102.552988] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 1102.828872] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 1104.503217] Key type lgssc registered [ 1105.491563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1112.029385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1112.041325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1113.365986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1117.182228] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1123.809601] LustreError: 39926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1123.833434] LustreError: 39926:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1124.229749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1124.641637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1128.493347] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1134.883909] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1140.532320] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1147.277713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1147.541079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1152.854669] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1158.907919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1159.882074] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1165.798533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1165.804526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1165.813083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1166.795513] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.28@tcp (stopping) [ 1167.739021] Lustre: server umount lustre-MDT0000 complete [ 1169.377140] LustreError: 40991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1169.378657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1169.387731] LustreError: 40991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1169.406492] Lustre: Skipped 1 previous similar message [ 1170.980188] LustreError: 39906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787377976 with bad export cookie 3785188167293016183 [ 1170.982262] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1170.991967] LustreError: 39906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1171.891257] LustreError: 41765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1171.919038] LustreError: 41765:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1171.929135] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.28@tcp (stopping) [ 1174.496384] LustreError: 39920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1174.501730] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1174.523262] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1177.382426] Lustre: server umount lustre-MDT0001 complete [ 1198.358302] Lustre: server umount lustre-OST0000 complete [ 1212.943627] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 1215.556814] Key type lgssc unregistered [ 1215.895222] LNet: 43780:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1215.904736] LNetError: 43780:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1215.918717] LNet: Removed LNI 192.168.206.128@tcp [ 1216.732194] Key type .llcrypt unregistered [ 1216.736960] Key type ._llcrypt unregistered [ 1239.404940] Key type ._llcrypt registered [ 1239.407235] Key type .llcrypt registered [ 1239.478961] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 1240.712423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1240.734466] alg: No test for adler32 (adler32-zlib) [ 1241.787233] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 1242.040441] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 1243.719175] Key type lgssc registered [ 1244.933720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1252.130647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1252.155720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1253.680950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1257.992511] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1264.098579] LustreError: 45198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1264.125162] LustreError: 45198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1264.989993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1265.308338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1269.372729] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1276.741633] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1282.303510] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1288.986765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1289.473810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1295.124397] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1298.691889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1302.129776] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1309.138189] Lustre: lustre-OST0000: Not available for connect from 192.168.206.28@tcp (stopping) [ 1309.157785] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1309.160631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1309.174781] Lustre: Skipped 1 previous similar message [ 1313.648818] Lustre: server umount lustre-OST0000 complete [ 1332.192031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1332.201401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1332.215956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1332.232401] Lustre: Skipped 2 previous similar messages [ 1334.753085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1334.754504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1334.780388] Lustre: Skipped 1 previous similar message [ 1336.869708] Lustre: server umount lustre-MDT0000 complete [ 1339.877706] LustreError: 46576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1339.905076] LustreError: 46576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1340.969420] LustreError: 45179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378146 with bad export cookie 2546209654186438977 [ 1341.010119] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1347.369554] Lustre: server umount lustre-MDT0001 complete [ 1358.156473] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 1360.594318] Key type lgssc unregistered [ 1360.898723] LNet: 48902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1360.914485] LNetError: 48902:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1360.939285] LNet: Removed LNI 192.168.206.128@tcp [ 1361.606907] Key type .llcrypt unregistered [ 1361.609190] Key type ._llcrypt unregistered [ 1382.654372] Key type ._llcrypt registered [ 1382.657314] Key type .llcrypt registered [ 1382.759471] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 1383.875420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1383.893036] alg: No test for adler32 (adler32-zlib) [ 1384.922227] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 1385.147580] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 1386.775219] Key type lgssc registered [ 1387.677864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1394.194149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1394.210884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1395.613726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1399.708172] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1405.921822] LustreError: 50301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1405.943105] LustreError: 50301:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1407.640806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1407.915306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1411.780262] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1418.749685] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1422.908485] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1428.010361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1428.279745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1434.425961] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1440.559536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1441.801923] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1448.003905] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1451.791149] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1463.948452] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-22 1h57m47s [ 1465.446965] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1469.410897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1469.423673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1469.444369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1470.417242] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.28@tcp (stopping) [ 1470.423480] Lustre: Skipped 2 previous similar messages [ 1471.477720] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.28@tcp (stopping) [ 1471.482582] Lustre: Skipped 3 previous similar messages [ 1473.902965] Lustre: server umount lustre-MDT0000 complete [ 1476.534810] LustreError: 52699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1477.123787] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1482.585391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1482.678795] LustreError: 50295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1482.696954] LustreError: 50295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1482.739470] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1482.898785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1482.942763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1486.517563] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1489.395820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1489.644570] Lustre: DEBUG MARKER: Wait for df (36505) ... [ 1490.860316] Lustre: DEBUG MARKER: done [ 1494.497032] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1494.505561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1494.517596] Lustre: Skipped 3 previous similar messages [ 1494.526276] Lustre: Skipped 1 previous similar message [ 1499.620022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1499.624627] Lustre: Skipped 1 previous similar message [ 1508.319150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1508.460989] Lustre: server umount lustre-OST0000 complete [ 1514.976730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1514.986474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1514.989994] Lustre: Skipped 1 previous similar message [ 1515.004466] Lustre: Skipped 3 previous similar messages [ 1518.497597] Lustre: server umount lustre-MDT0000 complete [ 1522.149185] LustreError: 50300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1522.170769] LustreError: 50300:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1522.248329] LustreError: 50281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378328 with bad export cookie 11164302585359587729 [ 1522.251282] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1522.265558] LustreError: 50281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1522.804252] Lustre: server umount lustre-MDT0001 complete [ 1535.364991] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 1538.467529] Key type lgssc unregistered [ 1538.847643] LNet: 55228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1538.856553] LNetError: 55228:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1538.880573] LNet: Removed LNI 192.168.206.128@tcp [ 1540.156677] Key type .llcrypt unregistered [ 1540.158953] Key type ._llcrypt unregistered [ 1577.573227] Key type ._llcrypt registered [ 1577.575876] Key type .llcrypt registered [ 1577.673641] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 1578.494931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1578.654994] alg: No test for adler32 (adler32-zlib) [ 1579.667340] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 1579.868098] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 1581.543742] Key type lgssc registered [ 1582.627405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1589.947220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1589.962281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1591.193830] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1591.218722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1591.395858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1595.958514] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1604.844067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.914667] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1604.945204] Lustre: 57439:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1605.273719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1609.787329] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1616.989354] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1622.338415] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1628.511884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1628.706261] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1628.912984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1634.543635] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1638.028559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1641.635891] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1648.416812] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-22 2h00m52s [ 1649.864486] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1657.501326] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1660.331377] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1660.554532] LustreError: 59195:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1680.888947] Lustre: lustre-MDT0000: Client 47cbd6dd-0088-4a04-8610-9b1252b9123a (at 192.168.206.28@tcp) reconnecting [ 1700.655293] LustreError: 59195:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1702.549188] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1704.505663] Lustre: DEBUG MARKER: done [ 1710.064169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1710.066472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1710.081404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1710.083393] Lustre: Skipped 1 previous similar message [ 1715.179796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1715.193603] Lustre: Skipped 2 previous similar messages [ 1715.955241] Lustre: server umount lustre-OST0000 complete [ 1720.288770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1720.306754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1720.317352] Lustre: Skipped 1 previous similar message [ 1726.385196] Lustre: server umount lustre-MDT0000 complete [ 1729.965234] LustreError: 60200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378535 with bad export cookie 2154616491545160378 [ 1729.967404] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1729.985550] LustreError: 60200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1730.349277] Lustre: server umount lustre-MDT0001 complete [ 1739.780083] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 1741.802811] Key type lgssc unregistered [ 1742.148217] LNet: 61781:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1742.154873] LNetError: 61781:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1742.178317] LNet: Removed LNI 192.168.206.128@tcp [ 1743.006860] Key type .llcrypt unregistered [ 1743.009808] Key type ._llcrypt unregistered [ 1783.219409] Key type ._llcrypt registered [ 1783.221272] Key type .llcrypt registered [ 1783.283584] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 1784.196607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1784.232201] alg: No test for adler32 (adler32-zlib) [ 1785.235475] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 1785.532667] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 1787.223422] Key type lgssc registered [ 1788.582703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1792.368130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1792.394148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1793.888675] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1793.902900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1794.016113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1795.349759] Lustre: Failing over lustre-MDT0000 [ 1795.997731] Lustre: server umount lustre-MDT0000 complete [ 1804.935958] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 02:03:29 (1787378609) [ 1811.672818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1812.058617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1816.376984] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1824.622292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1824.725128] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1824.759510] Lustre: 64581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1824.965226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1828.676787] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1834.707333] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1839.986812] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1846.304827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1846.480532] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1846.679573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1852.171962] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1856.677721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1858.521448] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1864.939838] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1872.352929] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1872.355813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1872.373177] Lustre: Skipped 2 previous similar messages [ 1872.381827] Lustre: Skipped 1 previous similar message [ 1877.476557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1877.492380] Lustre: Skipped 3 previous similar messages [ 1878.214465] Lustre: server umount lustre-MDT0000 complete [ 1882.364225] LustreError: 64573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378688 with bad export cookie 14598583793191706343 [ 1882.379251] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1882.383770] LustreError: 64573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1882.614120] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.748802] Lustre: server umount lustre-MDT0001 complete [ 1886.530926] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1891.113971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1900.265450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1908.214758] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xca98969d00f767ca [ 1908.239209] Lustre: MGC192.168.206.128@tcp: Connection restored to 0@lo (at 0@lo) [ 1908.613208] LustreError: 66428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1908.643285] LustreError: 66428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1908.734846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1908.899043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1912.811290] LustreError: 66429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1912.819419] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1914.277800] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1917.924667] LustreError: 66428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1922.499129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1922.923129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1923.949753] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1926.909263] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1934.184918] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1940.288323] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1947.461128] Lustre: server umount lustre-MDT0000 complete [ 1948.641198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1948.647460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1948.663748] LustreError: 66428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1950.934169] LustreError: 64573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378756 with bad export cookie 14598583793191708618 [ 1950.941696] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1950.949124] LustreError: 64573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1953.332319] Lustre: server umount lustre-MDT0001 complete [ 1959.594277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1968.748761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.167669] Lustre: 63505:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787378760/real 1787378760] req@000000005eb6f247 x1874202293501952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787378776 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1971.192171] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.206361] Lustre: Skipped 2 previous similar messages [ 1972.192709] Lustre: 63503:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787378761/real 1787378761] req@00000000d90cb237 x1874202293502080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787378777 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1975.624051] LustreError: 66428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1975.641247] LustreError: 66428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1975.731909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1975.796787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1976.352668] Lustre: 63504:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787378765/real 1787378765] req@00000000a92584fe x1874202293502208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787378781 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1980.895672] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1981.793351] Lustre: 63506:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787378771/real 1787378771] req@000000007de191ce x1874202293502592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787378787 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1981.843167] Lustre: 63506:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1985.952600] Lustre: 63503:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787378775/real 1787378775] req@00000000acbf696e x1874202293502848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787378791 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1987.046304] LustreError: 71032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1987.064824] LustreError: 71032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1990.121596] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1990.169978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1995.141721] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1995.787129] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2002.357344] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2007.499201] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2014.413718] Lustre: server umount lustre-MDT0000 complete [ 2016.229440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2016.239571] LustreError: 71593:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.253365] Lustre: Skipped 2 previous similar messages [ 2016.255139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2016.284872] LustreError: 71593:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2018.345493] LustreError: 64572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378824 with bad export cookie 14598583793191709997 [ 2018.351193] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2018.359702] LustreError: 64572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2018.918725] Lustre: server umount lustre-MDT0001 complete [ 2025.052868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2028.775181] Lustre: DEBUG MARKER: files compared the same [ 2040.831894] Lustre: server umount lustre-OST0000 complete [ 2057.403883] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 2060.011712] Key type lgssc unregistered [ 2060.308284] LNet: 74306:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2060.328773] LNetError: 74306:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2060.364493] LNet: Removed LNI 192.168.206.128@tcp [ 2061.331556] Key type .llcrypt unregistered [ 2061.342295] Key type ._llcrypt unregistered [ 2087.981545] Key type ._llcrypt registered [ 2087.984394] Key type .llcrypt registered [ 2088.089457] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 2089.538578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2089.585151] alg: No test for adler32 (adler32-zlib) [ 2090.727499] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 2090.923103] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 2092.599852] Key type lgssc registered [ 2093.776967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2102.027515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2102.039680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2103.583036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2107.981326] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug +malloc all [ 2114.016680] LustreError: 75725:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2114.048349] LustreError: 75725:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2116.958681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2117.224741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2121.397665] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug +malloc all [ 2128.272995] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2133.599100] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2140.332187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2140.601544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2146.879774] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug +malloc all [ 2148.852250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2154.897754] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2164.191755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2164.192885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.203132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2164.235179] Lustre: Skipped 1 previous similar message [ 2169.320917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2169.336857] Lustre: Skipped 2 previous similar messages [ 2174.432273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2174.439328] Lustre: Skipped 1 previous similar message [ 2176.991317] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2177.173911] Lustre: server umount lustre-OST0000 complete [ 2183.657064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2183.664308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2183.692505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2184.681296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2187.531213] Lustre: server umount lustre-MDT0000 complete [ 2189.793836] LustreError: 76453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2189.812096] LustreError: 76453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2191.445861] LustreError: 75705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378997 with bad export cookie 17760206968426643500 [ 2191.449558] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2191.459764] LustreError: 75705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2191.781931] Lustre: server umount lustre-MDT0001 complete [ 2203.394545] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 2206.352841] Key type lgssc unregistered [ 2206.694218] LNet: 79259:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2206.707690] LNetError: 79259:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2206.738878] LNet: Removed LNI 192.168.206.128@tcp [ 2207.704404] Key type .llcrypt unregistered [ 2207.706408] Key type ._llcrypt unregistered [ 2230.866285] Key type ._llcrypt registered [ 2230.868960] Key type .llcrypt registered [ 2231.025115] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 2232.312272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2232.432211] alg: No test for adler32 (adler32-zlib) [ 2233.607265] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 2233.899428] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 2235.587427] Key type lgssc registered [ 2236.812327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2244.149505] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2244.171232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2259.743396] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2264.863312] LustreError: 80649:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.128@tcp: failed processing log, type 4: rc = -110 [ 2291.551177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2298.167177] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2307.346488] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2316.521894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2316.671572] Lustre: *** cfs_fail_loc=706, val=0*** [ 2317.032603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2317.095506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2321.896288] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2322.401465] LustreError: 80675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2322.446471] LustreError: 80675:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2326.499847] LustreError: 80674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2326.525546] LustreError: 80674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2331.619060] LustreError: 81736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2331.654163] LustreError: 81736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2332.064640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2332.582145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2337.792178] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2345.419645] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2351.598363] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2358.241307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2358.259464] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2358.270684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2358.290166] Lustre: Skipped 1 previous similar message [ 2360.162820] Lustre: server umount lustre-OST0000 complete [ 2368.480704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2368.483129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2368.492078] Lustre: Skipped 1 previous similar message [ 2368.493171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2368.519248] Lustre: Skipped 1 previous similar message [ 2370.272645] Lustre: server umount lustre-MDT0000 complete [ 2373.275585] LustreError: 80657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379179 with bad export cookie 12841645450467499427 [ 2373.285405] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2373.289839] LustreError: 80657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2373.596862] Lustre: server umount lustre-MDT0001 complete [ 2383.825225] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 2386.962112] Key type lgssc unregistered [ 2387.281093] LNet: 84265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2387.294527] LNetError: 84265:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2387.330179] LNet: Removed LNI 192.168.206.128@tcp [ 2388.232435] Key type .llcrypt unregistered [ 2388.234427] Key type ._llcrypt unregistered [ 2412.477162] Key type ._llcrypt registered [ 2412.480895] Key type .llcrypt registered [ 2412.615298] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 2413.579620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2413.635923] alg: No test for adler32 (adler32-zlib) [ 2414.957424] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 2415.233308] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 2416.903220] Key type lgssc registered [ 2417.885826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2425.593923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2425.615520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2431.685930] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2441.608600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2442.579836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2447.844765] LustreError: 86374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.869254] LustreError: 86374:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2448.266131] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2452.963920] LustreError: 86378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.991392] LustreError: 86378:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2455.847760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2456.226448] LustreError: 86377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2456.252196] LustreError: 86377:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2456.271977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2461.673602] LustreError: 86378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2461.698934] LustreError: 86378:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2463.214509] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2466.784485] LustreError: 86377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2466.800106] LustreError: 86377:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2472.572574] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2472.589325] Lustre: MGS: Client 2abc0c5c-531b-45d2-9c43-864adaa47591 (at 0@lo) reconnecting [ 2472.618562] LustreError: 86388:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2472.841951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2472.966813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2478.152244] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2493.415600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2493.431651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2493.432818] LustreError: Skipped 1 previous similar message [ 2493.433491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2493.475795] Lustre: Skipped 1 previous similar message [ 2497.867103] Lustre: server umount lustre-OST0000 complete [ 2503.648760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2503.660882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2503.678128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2503.683043] Lustre: Skipped 2 previous similar messages [ 2508.087086] Lustre: server umount lustre-MDT0000 complete [ 2508.768286] LustreError: 86378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.807252] LustreError: 86378:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2512.845297] Lustre: server umount lustre-MDT0001 complete [ 2517.069726] Lustre: server umount MGS complete [ 2529.942958] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 02:15:33 (1787379333) [ 2564.956881] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 2575.217605] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 2614.643526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2620.257874] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2631.510113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.784479] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2631.837548] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2631.907663] Lustre: lustre-MDT0001: new disk, initializing [ 2631.985369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2632.027513] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2632.042643] Lustre: 93542:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2632.053567] LustreError: 93542:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2637.793273] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2649.845094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2650.176606] Lustre: lustre-OST0000: new disk, initializing [ 2650.183062] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2650.195247] Lustre: 94309:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2650.286170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2657.312147] Lustre: 94603:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000056597b9a x1874202954079872/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787379479 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2657.462630] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2668.467950] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2674.551218] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2674.569384] Lustre: MGS: Client a19426a7-835f-436b-a13e-4bb2d9f5bdf6 (at 0@lo) reconnecting [ 2674.647823] Lustre: 92810:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2674.675330] LustreError: 93528:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2674.690851] LustreError: 93528:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2674.704265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2674.765611] Lustre: lustre-MDT0000: new disk, initializing [ 2674.863806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2674.888229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2674.905477] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2675.936040] Lustre: 95379:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000002f32fd45 x1874202954086656/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787379497 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2678.551915] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2681.055633] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2684.905476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2684.924956] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2685.106402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2709.473849] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2709.479359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2709.486306] Lustre: Skipped 2 previous similar messages [ 2709.497169] Lustre: Skipped 1 previous similar message [ 2714.586427] Lustre: server umount lustre-OST0000 complete [ 2719.713452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2719.715903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2719.725039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2719.733067] Lustre: Skipped 1 previous similar message [ 2719.747886] Lustre: Skipped 1 previous similar message [ 2724.635108] Lustre: server umount lustre-MDT0000 complete [ 2724.840178] LustreError: 93518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2724.874961] LustreError: 93518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2735.419557] Lustre: server umount lustre-MDT0001 complete [ 2739.440548] Lustre: server umount MGS complete [ 2752.945545] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:19:17 (1787379557) [ 2772.512538] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 2775.898614] Key type lgssc unregistered [ 2776.250862] LNet: 97877:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2776.254619] LNetError: 97877:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2776.268208] LNet: Removed LNI 192.168.206.128@tcp [ 2777.510178] Key type .llcrypt unregistered [ 2777.512839] Key type ._llcrypt unregistered [ 2797.427762] Key type ._llcrypt registered [ 2797.434846] Key type .llcrypt registered [ 2797.629089] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 2799.171101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2799.221386] alg: No test for adler32 (adler32-zlib) [ 2800.452714] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 2800.704307] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 2802.479208] Key type lgssc registered [ 2803.895474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2812.171574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2812.191943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2813.756563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2818.317637] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2824.166000] LustreError: 99099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2824.179785] LustreError: 99099:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2827.422581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2827.783881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2832.750756] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2840.837159] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2846.692924] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2854.170860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2854.659901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2858.747277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2861.066498] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2868.677953] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2883.552471] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2883.557100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2883.567447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2883.601899] Lustre: Skipped 1 previous similar message [ 2886.350108] Lustre: server umount lustre-OST0000 complete [ 2893.795819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2893.808654] Lustre: Skipped 1 previous similar message [ 2893.809143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2893.828231] Lustre: Skipped 2 previous similar messages [ 2896.751796] Lustre: server umount lustre-MDT0000 complete [ 2898.912158] LustreError: 99100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2898.943462] LustreError: 99100:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2900.778421] LustreError: 99081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379706 with bad export cookie 16549847723833641459 [ 2900.790318] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2901.078831] Lustre: server umount lustre-MDT0001 complete [ 2912.163094] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 2915.242991] Key type lgssc unregistered [ 2915.674795] LNet: 102633:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2915.681016] LNetError: 102633:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2915.703611] LNet: Removed LNI 192.168.206.128@tcp [ 2916.519969] Key type .llcrypt unregistered [ 2916.522247] Key type ._llcrypt unregistered [ 2930.881893] Key type ._llcrypt registered [ 2930.884034] Key type .llcrypt registered [ 2930.997893] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 2931.830932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2931.842494] alg: No test for adler32 (adler32-zlib) [ 2932.876805] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 2933.209735] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 2934.888142] Key type lgssc registered [ 2936.132186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2942.485934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2942.490935] LustreError: 103707:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2942.528822] LustreError: 103711:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2942.534403] LustreError: 103707:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4964 [ 2942.539279] LustreError: 103707:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2942.552087] LustreError: 103707:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2942.558543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2944.139688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2951.894867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2952.362520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2957.320364] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2962.887985] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing lsmod [ 2966.089114] LustreError: 104916:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2966.092723] LustreError: 104919:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2966.104913] LustreError: 104916:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 2966.117650] LustreError: 104916:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2966.126287] LustreError: 104916:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2966.126863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2966.644065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2968.692069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2973.152671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2973.161528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2973.174243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2974.178402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2974.185076] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.198348] Lustre: Skipped 1 previous similar message [ 2974.220620] Lustre: Skipped 1 previous similar message [ 2975.626704] Lustre: server umount lustre-MDT0000 complete [ 2977.318454] LustreError: 103749:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2977.337053] LustreError: 103749:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2979.210994] LustreError: 103728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379785 with bad export cookie 18112643606891476479 [ 2979.236687] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2979.425383] LustreError: 104180:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003b21e3a1 x1874203496174720/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2979.696791] Lustre: server umount lustre-MDT0001 complete [ 2993.941140] Lustre: server umount lustre-OST0000 complete [ 3002.072905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3002.607200] LustreError: 105981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3002.696567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3007.077956] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3007.980738] LustreError: 105982:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3013.089904] LustreError: 105981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3017.055635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3017.522803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3021.740342] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3029.607022] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3034.999420] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3040.602263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3040.841648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3045.895508] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3048.046348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 3052.828558] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3062.256690] Lustre: server umount lustre-OST0000 complete [ 3068.898693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3068.899700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3068.915441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3068.931790] Lustre: Skipped 2 previous similar messages [ 3074.028952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3074.042568] Lustre: Skipped 1 previous similar message [ 3076.261465] Lustre: server umount lustre-MDT0000 complete [ 3079.140834] LustreError: 105981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3079.181853] LustreError: 105981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3080.101926] LustreError: 105962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379885 with bad export cookie 18112643606891477424 [ 3080.113914] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3080.126101] LustreError: 105962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3089.537493] Lustre: server umount lustre-MDT0001 complete [ 3098.919391] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 3101.216946] Key type lgssc unregistered [ 3101.417895] LNet: 109515:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3101.429835] LNetError: 109515:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3101.451149] LNet: Removed LNI 192.168.206.128@tcp [ 3102.090691] Key type .llcrypt unregistered [ 3102.092688] Key type ._llcrypt unregistered [ 3124.248716] Key type ._llcrypt registered [ 3124.251457] Key type .llcrypt registered [ 3124.356706] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 3125.220090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3125.256252] alg: No test for adler32 (adler32-zlib) [ 3126.571531] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 3126.837543] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 3128.551174] Key type lgssc registered [ 3129.988639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3138.217873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3138.233566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3139.723454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3144.181251] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3149.281847] LustreError: 110926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3149.320829] LustreError: 110926:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3153.403259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3153.727207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3157.994454] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3164.897748] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3169.849239] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3176.799082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3177.202771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3182.961401] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3184.390679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3191.289470] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3197.704257] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3201.921394] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3209.189494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3209.200488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3209.217471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3210.213714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3210.214510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3212.995527] Lustre: server umount lustre-OST0000 complete [ 3220.448112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3220.451894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3220.452260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3220.452265] Lustre: Skipped 1 previous similar message [ 3220.472478] Lustre: Skipped 1 previous similar message [ 3222.910402] Lustre: server umount lustre-MDT0000 complete [ 3225.578793] LustreError: 112166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3225.602741] LustreError: 112166:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3226.688849] LustreError: 110908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380032 with bad export cookie 1355599823982698974 [ 3226.694663] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3226.703582] LustreError: 110908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3227.103942] Lustre: server umount lustre-MDT0001 complete [ 3238.379875] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 3241.263048] Key type lgssc unregistered [ 3241.533577] LNet: 114566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3241.548777] LNetError: 114566:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3241.564307] LNet: Removed LNI 192.168.206.128@tcp [ 3242.416183] Key type .llcrypt unregistered [ 3242.419367] Key type ._llcrypt unregistered [ 3262.085483] Key type ._llcrypt registered [ 3262.089123] Key type .llcrypt registered [ 3262.249661] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 3263.344079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3263.367638] alg: No test for adler32 (adler32-zlib) [ 3264.641594] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 3264.908071] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 3266.591147] Key type lgssc registered [ 3267.740328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3275.605852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3275.635782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3277.268991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3282.778722] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3287.536588] LustreError: 115793:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3287.568776] LustreError: 115793:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3291.742417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3292.054763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3296.711439] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3304.142238] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3308.932560] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3315.708507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3316.068652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3322.023443] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3322.233957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3329.677432] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3342.817689] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3342.822103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3342.827833] LustreError: Skipped 1 previous similar message [ 3342.844342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3342.846965] Lustre: Skipped 1 previous similar message [ 3342.868587] Lustre: Skipped 1 previous similar message [ 3344.229329] Lustre: server umount lustre-OST0000 complete [ 3353.067473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3353.079063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3353.084906] Lustre: Skipped 1 previous similar message [ 3354.810162] Lustre: server umount lustre-MDT0000 complete [ 3358.177817] LustreError: 115794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3358.210508] LustreError: 115794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3358.656152] LustreError: 115773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380164 with bad export cookie 12662428814550458875 [ 3358.657535] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3358.678301] LustreError: 115773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3359.123499] Lustre: server umount lustre-MDT0001 complete [ 3370.870225] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 3373.715371] Key type lgssc unregistered [ 3373.977639] LNet: 119327:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3373.987646] LNetError: 119327:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3374.015917] LNet: Removed LNI 192.168.206.128@tcp [ 3374.893749] Key type .llcrypt unregistered [ 3374.896426] Key type ._llcrypt unregistered [ 3398.068055] Key type ._llcrypt registered [ 3398.071289] Key type .llcrypt registered [ 3398.193183] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 3399.160873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3399.187674] alg: No test for adler32 (adler32-zlib) [ 3400.330421] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 3400.518837] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 3402.231153] Key type lgssc registered [ 3403.672244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3411.724072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3411.740952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3413.311159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3418.308268] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3423.716071] LustreError: 120723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3423.762033] LustreError: 120723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3426.876180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3427.398701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3431.842292] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3441.107829] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3447.541971] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3456.413929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3456.905124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3458.995103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 3463.959886] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3471.044186] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3478.895869] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3482.924602] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3482.931805] Lustre: Skipped 2 previous similar messages [ 3484.352828] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3484.391804] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3489.336355] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3489.346448] Lustre: Skipped 2 previous similar messages [ 3494.566614] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3494.589572] Lustre: Skipped 1 previous similar message [ 3499.787871] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.28@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3499.797915] Lustre: Skipped 2 previous similar messages [ 3500.676396] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.28@tcp 192.168.0.[2,10]@tcp [ 3514.679739] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.28/24@tcp in log lustre-MDT0000 [ 3514.684956] Lustre: Skipped 2 previous similar messages [ 3520.640123] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.28/24@tcp [ 3520.650170] Lustre: Skipped 1 previous similar message [ 3527.147739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3527.148332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3527.154856] LustreError: Skipped 2 previous similar messages [ 3527.170788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3528.575133] Lustre: server umount lustre-OST0000 complete [ 3533.797249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3533.801879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3533.828308] Lustre: Skipped 1 previous similar message [ 3533.856637] Lustre: Skipped 1 previous similar message [ 3538.603984] Lustre: server umount lustre-MDT0000 complete [ 3538.913754] LustreError: 120722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3538.942136] LustreError: 120722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3542.393554] LustreError: 120703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380348 with bad export cookie 5662782627240240955 [ 3542.394410] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3542.400534] LustreError: 120703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3553.038788] Lustre: server umount lustre-MDT0001 complete [ 3564.185820] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 3567.472534] Key type lgssc unregistered [ 3567.736901] LNet: 125548:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3567.747226] LNetError: 125548:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3567.779512] LNet: Removed LNI 192.168.206.128@tcp [ 3568.657530] Key type .llcrypt unregistered [ 3568.660750] Key type ._llcrypt unregistered [ 3599.420686] Key type ._llcrypt registered [ 3599.423889] Key type .llcrypt registered [ 3599.550949] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 3600.693229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3600.744303] alg: No test for adler32 (adler32-zlib) [ 3601.773264] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 3601.985905] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 3603.687227] Key type lgssc registered [ 3604.958476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3612.332039] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3612.357569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3613.826705] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3613.872202] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.28/24@tcp [ 3613.875672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3618.418287] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3623.396047] LustreError: 127111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3623.421057] LustreError: 127111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3628.084355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3628.390817] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3628.422749] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.28/24@tcp [ 3628.548044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3633.758111] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3642.316057] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3648.351749] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3655.583498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3655.996214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3659.061203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 3662.684527] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3671.572271] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3683.812239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3683.839793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3683.843481] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3683.851786] Lustre: Skipped 1 previous similar message [ 3688.928549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3688.943127] Lustre: Skipped 1 previous similar message [ 3689.671863] Lustre: server umount lustre-OST0000 complete [ 3694.048136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3694.050394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3694.057599] Lustre: Skipped 2 previous similar messages [ 3699.169227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3699.178215] Lustre: Skipped 2 previous similar messages [ 3699.971913] Lustre: server umount lustre-MDT0000 complete [ 3703.674688] LustreError: 127093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380509 with bad export cookie 4450697818325380717 [ 3703.687735] LustreError: 127093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3703.689201] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3704.074769] Lustre: server umount lustre-MDT0001 complete [ 3715.071159] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 3717.749969] Key type lgssc unregistered [ 3718.080204] LNet: 130702:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3718.094222] LNetError: 130702:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3719.144989] LNet: Removed LNI 192.168.206.128@tcp [ 3720.108227] Key type .llcrypt unregistered [ 3720.110487] Key type ._llcrypt unregistered [ 3745.527932] Key type ._llcrypt registered [ 3745.530992] Key type .llcrypt registered [ 3745.666391] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 3760.597906] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 3761.423123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3761.547058] alg: No test for adler32 (adler32-zlib) [ 3762.676315] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 3762.923199] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 3764.575148] Key type lgssc registered [ 3765.655858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3802.519517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3802.535605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3803.765956] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3803.793227] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3803.866451] Lustre: lustre-MDT0000: new disk, initializing [ 3803.971260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3804.011303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3809.044785] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3819.231022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3819.343452] Lustre: 134101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3819.422461] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3819.429284] Lustre: Skipped 1 previous similar message [ 3819.512908] Lustre: lustre-MDT0001: new disk, initializing [ 3819.600842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3819.633511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3819.647636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3824.687470] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3834.856714] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3841.203607] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3847.198772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3847.424198] Lustre: lustre-OST0000: new disk, initializing [ 3847.427621] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3847.433444] Lustre: 136072:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3847.503289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3853.408527] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3855.982256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3856.006167] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3856.147836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3863.757482] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3869.088318] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3869.396311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3874.739277] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3874.969879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3881.952497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3881.964343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3881.967139] LustreError: Skipped 1 previous similar message [ 3881.972331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3881.999880] Lustre: Skipped 1 previous similar message [ 3887.075518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3887.083585] Lustre: Skipped 1 previous similar message [ 3891.167131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3891.418359] Lustre: server umount lustre-OST0000 complete [ 3896.289561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3896.306571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3896.329215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3896.333882] Lustre: Skipped 1 previous similar message [ 3897.313566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3901.779671] Lustre: server umount lustre-MDT0000 complete [ 3905.187401] LustreError: 134094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380710 with bad export cookie 12319159350817051825 [ 3905.201436] LustreError: 134094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3905.210563] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3905.651534] Lustre: server umount lustre-MDT0001 complete [ 3913.616155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3914.061230] LustreError: 138289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3914.094550] LustreError: 138289:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3914.171473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3918.299775] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3919.338292] LustreError: 138289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3923.427605] LustreError: 138290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3926.950569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3927.283420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3931.851318] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3939.473731] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3945.269947] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3951.413721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3951.712717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3956.980851] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3963.197827] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3967.802203] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3968.012836] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3972.313145] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3972.484827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3981.641442] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3981.778247] Lustre: lustre-OST0001: new disk, initializing [ 3981.781702] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3981.786884] Lustre: 141660:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3981.885230] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3987.556804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3987.574787] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3987.725738] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3988.822583] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3997.401045] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3997.595533] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4002.059966] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4002.261978] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4008.103913] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff942488a10000.ost_server_uuid 50 [ 4009.582675] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff942488a10000.ost_server_uuid in FULL state after 0 sec [ 4023.781803] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4023.782257] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4023.790447] LustreError: Skipped 1 previous similar message [ 4023.791991] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4023.792014] Lustre: Skipped 2 previous similar messages [ 4023.814387] Lustre: Skipped 1 previous similar message [ 4024.897828] Lustre: server umount lustre-OST0001 complete [ 4028.899571] LustreError: 140111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4028.901587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4028.923682] LustreError: 140111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4028.950772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4028.957054] Lustre: Skipped 2 previous similar messages [ 4034.021950] LustreError: 140112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4034.040416] LustreError: 140112:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4034.827252] Lustre: server umount lustre-OST0000 complete [ 4039.136834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4039.145822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4039.157378] Lustre: Skipped 2 previous similar messages [ 4039.168540] Lustre: Skipped 2 previous similar messages [ 4044.858700] Lustre: server umount lustre-MDT0000 complete [ 4048.889611] LustreError: 138271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380854 with bad export cookie 12319159350817052707 [ 4048.893171] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4048.905310] LustreError: 138271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4049.222711] Lustre: server umount lustre-MDT0001 complete [ 4068.448058] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 4071.554361] Key type lgssc unregistered [ 4071.847538] LNet: 144661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4071.852957] LNetError: 144661:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4071.873135] LNet: Removed LNI 192.168.206.128@tcp [ 4073.067647] Key type .llcrypt unregistered [ 4073.073471] Key type ._llcrypt unregistered [ 4113.667960] Key type ._llcrypt registered [ 4113.670986] Key type .llcrypt registered [ 4113.764719] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 4127.736554] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 4128.663553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4128.696461] alg: No test for adler32 (adler32-zlib) [ 4129.853412] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 4130.119093] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 4131.775437] Key type lgssc registered [ 4132.754291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4170.455614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4170.516696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4171.842804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4171.893869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4171.994873] Lustre: lustre-MDT0000: new disk, initializing [ 4172.076371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4172.089701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4176.620314] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4186.247620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4186.336634] Lustre: 148815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4186.410374] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4186.415338] Lustre: Skipped 1 previous similar message [ 4186.476600] Lustre: lustre-MDT0001: new disk, initializing [ 4186.552034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4186.593996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4186.608859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4190.905813] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4200.230777] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4205.285979] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4211.499406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4211.767817] Lustre: lustre-OST0000: new disk, initializing [ 4211.772096] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4211.777523] Lustre: 150783:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4211.824601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4218.987768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4219.019477] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4219.128294] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4219.147498] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4228.768566] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4233.011298] Lustre: Failing over lustre-OST0000 [ 4234.726363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4234.728100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4234.730253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4234.745298] Lustre: Skipped 1 previous similar message [ 4234.763748] Lustre: Skipped 1 previous similar message [ 4235.236652] Lustre: server umount lustre-OST0000 complete [ 4253.232614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4253.469687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4253.480798] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4254.501674] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4259.188584] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4262.429659] Lustre: lustre-OST0000: Recovery over after 0:08, of 3 clients 3 recovered and 0 were evicted. [ 4262.437073] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4263.438077] Lustre: Failing over lustre-MDT0000 [ 4263.698777] Lustre: server umount lustre-MDT0000 complete [ 4264.931417] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4264.949338] Lustre: Skipped 1 previous similar message [ 4270.055118] LustreError: 149704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4270.073238] LustreError: 149704:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4272.563424] LustreError: 148822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4275.168883] LustreError: 148820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4275.184510] LustreError: 148820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4277.682809] LustreError: 148821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4280.287862] Lustre: 147036:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787381070/real 1787381070] req@00000000d950c0a6 x1874204751344896/t0(0) o400->MGC192.168.206.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787381086 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4280.311521] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4282.290749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4282.805173] LustreError: 148821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4282.818330] LustreError: 148821:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4290.533352] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x48b6b714f121816d [ 4290.552320] Lustre: MGC192.168.206.128@tcp: Connection restored to 0@lo (at 0@lo) [ 4290.563790] Lustre: Skipped 1 previous similar message [ 4290.810816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4290.858253] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4293.044142] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4294.732897] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4296.171622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4296.206735] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4296.246035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4301.793633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4301.804440] LustreError: Skipped 1 previous similar message [ 4301.810838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4301.824796] Lustre: Skipped 1 previous similar message [ 4301.830930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4303.328985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4303.341740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4306.403522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4306.417163] Lustre: Skipped 1 previous similar message [ 4307.655913] Lustre: server umount lustre-OST0000 complete [ 4316.648295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4316.655177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4316.675415] Lustre: Skipped 1 previous similar message [ 4316.688401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4316.702633] Lustre: Skipped 1 previous similar message [ 4317.960928] Lustre: server umount lustre-MDT0000 complete [ 4321.558564] LustreError: 150782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381127 with bad export cookie 5239576517064884589 [ 4321.564973] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4321.575327] LustreError: 150782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4322.056963] Lustre: server umount lustre-MDT0001 complete [ 4334.642634] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 4337.138535] Key type lgssc unregistered [ 4337.384498] LNet: 154441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4337.389280] LNetError: 154441:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4337.412300] LNet: Removed LNI 192.168.206.128@tcp [ 4338.127504] Key type .llcrypt unregistered [ 4338.131573] Key type ._llcrypt unregistered [ 4372.963733] Key type ._llcrypt registered [ 4372.966371] Key type .llcrypt registered [ 4373.044647] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 4374.196877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4374.272201] alg: No test for adler32 (adler32-zlib) [ 4375.277493] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 4375.474658] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 4377.159179] Key type lgssc registered [ 4378.433974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4386.949788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4386.967581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4388.330636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4392.597415] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4398.563154] LustreError: 155962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4398.594681] LustreError: 155962:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4401.001626] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4403.691763] LustreError: 155961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4404.591141] LustreError: 155990:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4404.597867] LustreError: 155990:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4404.602433] LustreError: 155990:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 4404.969262] Lustre: server umount lustre-MDT0000 complete [ 4418.265284] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 4426.856633] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 4461.385169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4461.715694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4461.756330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4461.850694] Lustre: lustre-MDT0000: new disk, initializing [ 4461.933287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4461.956909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4465.869088] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4476.176041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4476.310286] Lustre: 159308:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4476.361092] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4476.364541] Lustre: Skipped 1 previous similar message [ 4476.417767] Lustre: lustre-MDT0001: new disk, initializing [ 4476.497711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4476.543740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4476.561482] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4480.589742] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4490.605271] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4496.689431] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4504.364240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4504.743799] Lustre: lustre-OST0000: new disk, initializing [ 4504.751614] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4504.760537] Lustre: 161276:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4504.865485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4510.800833] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4514.457079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4514.472629] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4514.618853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4520.902624] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4525.228151] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4527.481164] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:48:51 (1787381331) [ 4783.589954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4783.599177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4783.624542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4786.143771] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4786.144415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4786.161428] Lustre: Skipped 1 previous similar message [ 4786.173475] Lustre: Skipped 2 previous similar messages [ 4787.985061] Lustre: server umount lustre-MDT0000 complete [ 4791.265773] LustreError: 159319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4791.277419] LustreError: 159319:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4791.599094] LustreError: 161275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381597 with bad export cookie 11041726984662599051 [ 4791.601315] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4791.609554] LustreError: 161275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4796.383820] LustreError: 159314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4796.384477] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4796.429047] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4798.074451] Lustre: server umount lustre-MDT0001 complete [ 4816.351117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4816.489493] Lustre: server umount lustre-OST0000 complete [ 4824.752316] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 4833.626672] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 4870.189897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4870.398943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4870.422920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4870.528703] Lustre: lustre-MDT0000: new disk, initializing [ 4870.635367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4870.675475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4874.815818] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4884.059291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4884.132145] Lustre: 165940:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4884.201827] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4884.202803] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4884.210079] Lustre: Skipped 1 previous similar message [ 4884.262200] Lustre: lustre-MDT0001: new disk, initializing [ 4884.335595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4884.362151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4884.370898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4887.569541] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4896.977781] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4902.300464] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4908.589415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4908.967549] Lustre: lustre-OST0000: new disk, initializing [ 4908.971234] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4908.977159] Lustre: 167910:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4909.084526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4915.293460] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4918.355475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4918.376408] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4918.520851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4924.516469] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4930.573985] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4930.939121] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4935.943132] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4936.255181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4939.246853] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4939.262966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4939.286458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4944.719295] Lustre: server umount lustre-OST0000 complete [ 4949.472828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4949.475647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4949.492690] Lustre: Skipped 2 previous similar messages [ 4949.495404] Lustre: Skipped 3 previous similar messages [ 4954.438854] Lustre: server umount lustre-MDT0000 complete [ 4954.601776] LustreError: 167918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4954.622208] LustreError: 167918:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4957.952453] LustreError: 166749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381763 with bad export cookie 11041726984662757881 [ 4957.955276] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4957.964194] LustreError: 166749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4958.304562] Lustre: server umount lustre-MDT0001 complete [ 4966.078348] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:56:10 (1787381770) [ 4982.549939] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 4992.764953] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 5030.706784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5031.070225] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5031.127793] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5031.243456] Lustre: lustre-MDT0000: new disk, initializing [ 5031.343035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5031.379458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5035.979802] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5046.966479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5047.175370] Lustre: 172665:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5047.555769] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5053.025200] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5062.940919] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5069.190873] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5077.303826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5077.689901] Lustre: lustre-OST0000: new disk, initializing [ 5077.693914] Lustre: Skipped 1 previous similar message [ 5077.698325] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5077.702604] Lustre: Skipped 2 previous similar messages [ 5077.713061] Lustre: 174642:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5084.739708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5084.751719] Lustre: Skipped 1 previous similar message [ 5084.760867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5084.940634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5084.992174] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5094.406962] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5105.632541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5105.636655] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5105.644857] LustreError: Skipped 2 previous similar messages [ 5105.645861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5105.645866] Lustre: Skipped 1 previous similar message [ 5105.672586] Lustre: Skipped 1 previous similar message [ 5116.384665] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5116.640574] Lustre: server umount lustre-OST0000 complete [ 5135.327514] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5135.627476] Lustre: server umount lustre-MDT0000 complete [ 5136.354845] LustreError: 175319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5136.377673] LustreError: 175319:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5139.763454] LustreError: 172659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381945 with bad export cookie 11041726984662758763 [ 5139.770439] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5139.793669] LustreError: 172659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5140.273532] Lustre: server umount lustre-MDT0001 complete [ 5149.877916] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:59:14 (1787381954) [ 5167.902826] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 5177.310774] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 5216.901969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5217.190574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5217.231582] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5217.362216] Lustre: lustre-MDT0000: new disk, initializing [ 5217.453525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5217.459205] Lustre: Skipped 2 previous similar messages [ 5217.477410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5222.359282] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5234.091374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5234.205762] Lustre: 178974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5234.607178] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5240.008925] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5250.117154] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5256.545805] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5264.706808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5265.169715] Lustre: 180962:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5272.821224] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5274.183425] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5274.316201] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5283.341529] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5295.073591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5295.079909] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5295.086811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5295.093026] Lustre: Skipped 3 previous similar messages [ 5295.107299] Lustre: Skipped 12 previous similar messages [ 5297.293051] Lustre: server umount lustre-OST0000 complete [ 5307.926350] Lustre: server umount lustre-MDT0000 complete [ 5310.436393] LustreError: 178985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5311.395481] LustreError: 178965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382117 with bad export cookie 11041726984662759848 [ 5311.406067] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5311.406437] LustreError: 178965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5311.852307] Lustre: server umount lustre-MDT0001 complete [ 5324.159571] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 5327.573342] Key type lgssc unregistered [ 5328.116731] LNet: 182819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5328.137977] LNetError: 182819:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5328.165252] LNet: Removed LNI 192.168.206.128@tcp [ 5329.362342] Key type .llcrypt unregistered [ 5329.364097] Key type ._llcrypt unregistered [ 5356.822488] Key type ._llcrypt registered [ 5356.825731] Key type .llcrypt registered [ 5356.973401] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 5358.448972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5358.821202] alg: No test for adler32 (adler32-zlib) [ 5360.099864] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 5360.369966] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 5362.031188] Key type lgssc registered [ 5363.435852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5371.718485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5371.734469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5373.252354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5377.543578] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5382.629148] LustreError: 184234:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5382.646402] LustreError: 184234:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5386.971475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5387.424716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5392.511211] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5401.306317] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5407.534728] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5414.814314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5415.151489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5418.230134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5422.677312] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5430.987668] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5453.287585] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5453.290985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5453.296471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5453.317076] Lustre: Skipped 1 previous similar message [ 5455.643056] Lustre: server umount lustre-OST0000 complete [ 5463.523472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5463.537625] Lustre: Skipped 1 previous similar message [ 5463.544687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5463.552300] Lustre: Skipped 1 previous similar message [ 5465.746785] Lustre: server umount lustre-MDT0000 complete [ 5468.642513] LustreError: 184233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5468.656883] LustreError: 184233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5469.661469] LustreError: 184216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382275 with bad export cookie 1519263313315611264 [ 5469.675177] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5469.677455] LustreError: 184216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5470.136726] Lustre: server umount lustre-MDT0001 complete [ 5484.081577] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 5486.717424] Key type lgssc unregistered [ 5486.986670] LNet: 187789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5486.992982] LNetError: 187789:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5487.012761] LNet: Removed LNI 192.168.206.128@tcp [ 5488.138889] Key type .llcrypt unregistered [ 5488.140995] Key type ._llcrypt unregistered [ 5513.537829] Key type ._llcrypt registered [ 5513.540691] Key type .llcrypt registered [ 5513.651715] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 5514.752093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5514.898987] alg: No test for adler32 (adler32-zlib) [ 5516.225667] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 5516.501212] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 5518.239196] Key type lgssc registered [ 5519.321678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5526.970965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5526.993194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5528.475205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5533.248690] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5538.785557] LustreError: 189207:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5538.805759] LustreError: 189207:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5541.404173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5541.758386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5546.217545] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5553.438908] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5558.722962] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5565.215170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5565.612645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5571.618670] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5573.809089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5579.832890] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5587.426787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5587.433833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5587.449276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5589.475523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5589.480177] Lustre: Skipped 1 previous similar message [ 5589.484456] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5589.503423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5593.017219] Lustre: lustre-OST0000: Not available for connect from 192.168.206.28@tcp (stopping) [ 5593.026470] Lustre: Skipped 1 previous similar message [ 5598.139339] Lustre: lustre-OST0000: Not available for connect from 192.168.206.28@tcp (stopping) [ 5598.158092] Lustre: Skipped 2 previous similar messages [ 5601.247145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5601.467403] Lustre: server umount lustre-OST0000 complete [ 5608.238663] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff94248341f000.ost_server_uuid 50 [ 5609.823341] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5623.776431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5623.787469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5623.800975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5623.807143] Lustre: Skipped 2 previous similar messages [ 5629.926758] Lustre: server umount lustre-MDT0000 complete [ 5633.777097] LustreError: 192255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382439 with bad export cookie 17298849943130015316 [ 5633.784973] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5633.793854] LustreError: 192255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5634.243364] Lustre: server umount lustre-MDT0001 complete [ 5645.079489] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 03:07:29 (1787382449) [ 5652.866333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5653.438658] LustreError: 193129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5653.538153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5658.612935] LustreError: 193130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5658.801982] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5663.716404] LustreError: 193129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5667.169222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5667.491637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5671.679000] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5679.284162] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5685.596219] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5693.025875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5693.437742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5698.547606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5700.273300] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5708.193260] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5715.282690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5715.531659] Lustre: lustre-OST0001: new disk, initializing [ 5715.533931] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5715.537857] Lustre: 195973:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5715.673579] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5721.959097] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5725.275410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5725.285833] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5725.370128] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5732.310129] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5735.912508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5735.928433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5735.940020] Lustre: Skipped 2 previous similar messages [ 5735.965144] Lustre: Skipped 5 previous similar messages [ 5741.725803] Lustre: server umount lustre-OST0000 complete [ 5743.545322] LustreError: 195828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5749.314533] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5749.544892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5753.783075] LustreError: 195828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5753.811488] LustreError: 195828:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5755.123304] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5755.369894] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5771.754921] LustreError: 194950:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5771.757404] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5771.758255] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5771.760553] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5771.760559] Lustre: Skipped 3 previous similar messages [ 5771.783319] LustreError: 194950:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 5771.849767] Lustre: Skipped 1 previous similar message [ 5774.949463] Lustre: server umount lustre-OST0001 complete [ 5780.448571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5780.455073] LustreError: Skipped 1 previous similar message [ 5780.458056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5784.421418] Lustre: server umount lustre-MDT0000 complete [ 5787.594544] LustreError: 194972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382593 with bad export cookie 17298849943130016387 [ 5787.596900] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5787.609396] LustreError: 194972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5787.924315] Lustre: server umount lustre-MDT0001 complete [ 5810.268609] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 03:10:14 (1787382614) [ 5817.226347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5817.536412] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5817.558486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5817.781941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5821.975625] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5831.767631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5831.956816] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5832.043908] Lustre: 199414:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5838.110173] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5847.290739] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5853.329026] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5861.095470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5861.284287] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5868.858773] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5869.764217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5877.875474] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5885.495405] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5885.555108] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5885.702488] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5885.708687] Lustre: Skipped 2 previous similar messages [ 5892.317619] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5892.605768] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5901.630697] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5905.379978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5905.405915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5905.424953] Lustre: Skipped 1 previous similar message [ 5905.433844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5905.440759] Lustre: Skipped 4 previous similar messages [ 5911.148394] Lustre: server umount lustre-OST0000 complete [ 5913.010919] LustreError: 202619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5913.045697] LustreError: 202619:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5928.416238] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5928.430963] LustreError: Skipped 1 previous similar message [ 5934.091899] Lustre: server umount lustre-OST0001 complete [ 5938.657121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5938.677080] Lustre: Skipped 3 previous similar messages [ 5944.536684] Lustre: server umount lustre-MDT0000 complete [ 5948.311923] LustreError: 200159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382754 with bad export cookie 17298849943130017794 [ 5948.315546] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5948.342041] LustreError: 200159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5948.979793] Lustre: server umount lustre-MDT0001 complete [ 5976.563381] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 03:13:00 (1787382780) [ 5994.663830] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 6006.838772] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 6047.681141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6047.992814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6048.035131] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6048.167795] Lustre: lustre-MDT0000: new disk, initializing [ 6048.248774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6048.277386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6052.893137] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6064.704303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6064.798465] Lustre: 207516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6064.882888] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6064.890035] Lustre: Skipped 1 previous similar message [ 6065.002807] Lustre: lustre-MDT0001: new disk, initializing [ 6065.155854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6065.178893] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6070.361169] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6081.444233] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6088.149092] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6095.813884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6096.366460] Lustre: lustre-OST0000: new disk, initializing [ 6096.378208] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6096.387633] Lustre: 209484:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6103.638476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6103.654657] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6103.749966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6104.187408] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6116.431979] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6121.679780] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6121.959685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6127.848536] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6128.081098] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6134.753482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6134.763926] LustreError: Skipped 1 previous similar message [ 6134.766467] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6134.777942] Lustre: Skipped 1 previous similar message [ 6134.783718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6134.789708] Lustre: Skipped 13 previous similar messages [ 6136.068422] Lustre: server umount lustre-OST0000 complete [ 6146.929634] Lustre: server umount lustre-MDT0000 complete [ 6150.126616] LustreError: 207521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6150.166289] LustreError: 207521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 6151.031262] LustreError: 209483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382956 with bad export cookie 17298849943130019579 [ 6151.038067] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6151.039829] LustreError: 209483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6151.374486] Lustre: server umount lustre-MDT0001 complete [ 6158.942497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6163.416984] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6171.284800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6176.298294] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6183.774782] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6190.030732] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6196.531781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6202.582067] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6209.099585] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6213.838758] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6214.103108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6218.665881] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6218.879529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6222.816092] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6222.826853] LustreError: Skipped 2 previous similar messages [ 6227.191831] Lustre: server umount lustre-OST0000 complete [ 6233.535529] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6237.975503] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6266.464238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6273.279371] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6274.939079] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6274.962653] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6281.066556] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6285.949177] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6289.333116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6289.350289] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6290.401291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6290.421484] Lustre: Skipped 5 previous similar messages [ 6290.431071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6290.437770] Lustre: Skipped 6 previous similar messages [ 6296.114782] Lustre: server umount lustre-OST0000 complete [ 6306.247059] Lustre: server umount lustre-MDT0000 complete [ 6309.596880] LustreError: 212436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383115 with bad export cookie 17298849943130020461 [ 6309.599181] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6309.603463] LustreError: 212436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6310.041411] Lustre: server umount lustre-MDT0001 complete [ 6317.512844] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 03:18:42 (1787383122) [ 6323.668145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6324.130242] LustreError: 217665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6324.151744] LustreError: 217665:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6324.194219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6324.202744] Lustre: Skipped 6 previous similar messages [ 6328.773702] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6337.179369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6341.841899] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6348.799784] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6354.079945] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6360.719884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6366.750214] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6373.487145] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6377.689618] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6377.859400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6382.061517] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6382.254240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6388.315362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6388.499898] Lustre: lustre-OST0001: new disk, initializing [ 6388.505287] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6388.513582] Lustre: 221036:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6393.966144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6393.981451] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6394.067422] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6394.349543] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6404.107567] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6408.184543] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6408.395656] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6412.445956] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6412.668437] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6414.820130] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6414.827184] LustreError: Skipped 4 previous similar messages [ 6420.835978] Lustre: server umount lustre-OST0001 complete [ 6427.837789] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6428.064668] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6433.376754] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6433.633124] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6463.298374] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6468.593932] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6468.605673] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6468.606800] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6469.727536] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6477.827948] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6479.333420] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6487.450605] Lustre: server umount lustre-OST0001 complete [ 6512.042847] Lustre: server umount lustre-OST0000 complete [ 6522.575434] Lustre: server umount lustre-MDT0000 complete [ 6526.366552] LustreError: 217647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383332 with bad export cookie 17298849943130021427 [ 6526.379204] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6526.388189] LustreError: 217647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6532.875561] Lustre: server umount lustre-MDT0001 complete [ 6558.102832] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:22:42 (1787383362) [ 6565.957736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6566.380691] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6566.425816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6572.299395] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6581.608272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6581.755319] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6581.809114] Lustre: 226282:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6586.808409] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6594.815754] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6601.787570] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6610.634571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6610.827225] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6618.945697] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6627.919117] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6637.014745] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6637.104579] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6643.707195] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6651.534994] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6653.303875] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6658.769988] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6659.088708] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6664.346410] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6664.645297] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6670.097125] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff942484ed0000.ost_server_uuid 50 [ 6672.210929] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff942484ed0000.ost_server_uuid in IDLE state after 0 sec [ 6673.559749] Lustre: Permanently deactivating lustre-OST0001 [ 6673.590723] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6679.167497] Lustre: Permanently reactivating lustre-OST0001 [ 6679.206051] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6679.216174] Lustre: Skipped 2 previous similar messages [ 6685.164052] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6685.171385] LustreError: Skipped 6 previous similar messages [ 6685.175317] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6685.185063] Lustre: Skipped 11 previous similar messages [ 6685.191179] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6685.196503] Lustre: Skipped 17 previous similar messages [ 6690.566588] Lustre: server umount lustre-OST0001 complete [ 6693.859996] LustreError: 228129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6693.867276] LustreError: 228129:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 36 previous similar messages [ 6699.957407] Lustre: server umount lustre-OST0000 complete [ 6709.974530] Lustre: server umount lustre-MDT0000 complete [ 6713.688325] LustreError: 227022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383519 with bad export cookie 17298849943130022715 [ 6713.691462] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6713.704795] LustreError: 227022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6713.982692] Lustre: server umount lustre-MDT0001 complete [ 6738.076353] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:25:42 (1787383542) [ 6746.849388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6747.160214] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6747.197348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6747.206222] Lustre: Skipped 2 previous similar messages [ 6752.082517] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6759.274804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6759.362140] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6759.411227] Lustre: 232978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6764.397735] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6771.551897] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6777.017383] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6783.764052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6783.972289] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6783.999844] Lustre: Permanently deactivating lustre-OST0000 [ 6784.012196] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6791.143116] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6792.232780] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6798.583512] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6804.536710] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6804.669100] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6811.596662] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6814.902827] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6821.156537] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6824.008716] Lustre: Permanently reactivating lustre-OST0000 [ 6824.036966] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6824.042723] Lustre: Skipped 2 previous similar messages [ 6831.724329] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6831.725891] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 39 seconds. I think it's dead, and I am evicting it. exp 00000000678598c1, cur 1787383637 deadline 1787383628 last 1787383598 [ 6831.738315] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6831.779729] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6831.790521] Lustre: Skipped 1 previous similar message [ 6831.860414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6843.936598] Lustre: server umount lustre-OST0001 complete [ 6853.974334] Lustre: server umount lustre-OST0000 complete [ 6864.741901] Lustre: server umount lustre-MDT0000 complete [ 6868.958650] LustreError: 232970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383674 with bad export cookie 17298849943130024388 [ 6868.963447] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6868.978032] LustreError: 232970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6869.533061] Lustre: server umount lustre-MDT0001 complete [ 6881.119294] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 6884.156938] Key type lgssc unregistered [ 6884.529040] LNet: 237858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6884.535341] LNetError: 237858:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6884.553666] LNet: Removed LNI 192.168.206.128@tcp [ 6885.519879] Key type .llcrypt unregistered [ 6885.522227] Key type ._llcrypt unregistered [ 6909.759287] Key type ._llcrypt registered [ 6909.763254] Key type .llcrypt registered [ 6909.918781] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 6911.707067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6911.899518] alg: No test for adler32 (adler32-zlib) [ 6913.032287] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 6913.333080] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 6915.007203] Key type lgssc registered [ 6916.443980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6927.324606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6927.341215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6928.855566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6933.560425] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6939.105400] LustreError: 239326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6939.128096] LustreError: 239326:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6941.998233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6942.204173] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6942.244870] Lustre: 239316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6942.264775] Lustre: Permanently deactivating lustre-MDT0001 [ 6942.294482] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6942.701398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6947.439051] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6952.480477] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6956.678258] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6963.607055] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6971.614684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6971.894470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6973.937938] LustreError: 241276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6973.966404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6979.045324] LustreError: 241276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6979.507132] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6984.166001] LustreError: 241168:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6988.152810] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6988.262825] LustreError: 241168:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6988.286111] LustreError: 241168:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6995.546367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6995.699429] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7001.089528] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7001.108422] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7002.258250] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7010.466776] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7014.619656] Lustre: Permanently reactivating lustre-MDT0001 [ 7014.664755] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7014.677850] Lustre: Skipped 2 previous similar messages [ 7016.732090] Lustre: lustre-MDT0001: Client d799c64e-4063-4bcd-bc12-0743eee2b792 (at 192.168.206.28@tcp) reconnecting [ 7020.111868] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7020.135537] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7020.149211] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7020.174063] LustreError: 243031:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7020.177406] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7020.193219] LustreError: 243031:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7020.193329] LustreError: 243031:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.128@tcp: cfg command failed: rc = -108 [ 7020.194145] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7020.194145] [ 7020.195240] LustreError: 239318:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7022.512732] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7024.116480] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7029.557201] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7029.790670] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7034.288371] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7034.497910] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7037.010367] Lustre: Permanently deactivating lustre-MDT0001 [ 7037.023759] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7037.031250] Lustre: Skipped 2 previous similar messages [ 7043.713986] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7052.260649] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7052.271375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7052.275717] Lustre: Skipped 3 previous similar messages [ 7052.285742] Lustre: Skipped 3 previous similar messages [ 7053.178935] Lustre: server umount lustre-MDT0000 complete [ 7056.899185] LustreError: 242878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383862 with bad export cookie 4482672540447912152 [ 7056.907945] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7056.915438] LustreError: 242878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7057.221170] Lustre: server umount lustre-MDT0001 complete [ 7067.060135] Lustre: server umount lustre-OST0000 complete [ 7078.583822] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:31:22 (1787383882) [ 7089.443097] Lustre: server umount lustre-OST0001 complete [ 7096.361962] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 7107.970867] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 7145.059184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7145.245583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7145.258273] Lustre: Skipped 2 previous similar messages [ 7145.291546] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7145.400328] Lustre: lustre-MDT0000: new disk, initializing [ 7145.482757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7145.497970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7149.757221] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7158.644452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7158.750507] Lustre: 247589:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7158.794596] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7158.797767] Lustre: Skipped 1 previous similar message [ 7158.869691] Lustre: lustre-MDT0001: new disk, initializing [ 7158.949504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7158.985948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7159.010306] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7162.260362] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7170.435019] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7175.576401] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7182.692622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7182.998833] Lustre: lustre-OST0000: new disk, initializing [ 7183.005597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7183.011647] Lustre: 249555:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7183.113908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7189.315967] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7191.117172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7191.125549] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7191.239979] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7198.669651] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7201.170913] LustreError: 247598:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7209.372747] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7209.529568] Lustre: lustre-OST0001: new disk, initializing [ 7209.535693] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7209.545805] Lustre: 250674:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7211.239158] LustreError: 247598:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7211.265401] LustreError: 247596:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7215.657495] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7219.259851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7219.271355] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7219.370169] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7221.346898] LustreError: 247596:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7225.101320] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7227.048742] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7229.934118] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7229.940045] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7229.963151] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7229.995855] Lustre: Skipped 1 previous similar message [ 7235.059757] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7235.065563] Lustre: Skipped 1 previous similar message [ 7235.461558] Lustre: server umount lustre-OST0001 complete [ 7240.168616] LustreError: 249539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7240.187067] LustreError: 249539:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7252.963089] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7252.963443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7252.969049] LustreError: Skipped 1 previous similar message [ 7252.970371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7252.985774] Lustre: Skipped 2 previous similar messages [ 7253.015280] Lustre: Skipped 1 previous similar message [ 7258.453876] Lustre: server umount lustre-OST0000 complete [ 7265.763559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7265.772312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7265.779936] Lustre: Skipped 1 previous similar message [ 7265.794875] Lustre: Skipped 3 previous similar messages [ 7268.304914] Lustre: server umount lustre-MDT0000 complete [ 7270.884128] LustreError: 247596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7270.909432] LustreError: 247596:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 7271.518299] LustreError: 247582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384077 with bad export cookie 4482672540447914749 [ 7271.530264] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7271.535579] LustreError: 247582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7278.043885] Lustre: server umount lustre-MDT0001 complete [ 7289.927873] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 7292.751384] Key type lgssc unregistered [ 7293.064911] LNet: 252711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7293.073649] LNetError: 252711:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7293.102647] LNet: Removed LNI 192.168.206.128@tcp [ 7294.013385] Key type .llcrypt unregistered [ 7294.017400] Key type ._llcrypt unregistered [ 7333.126739] Key type ._llcrypt registered [ 7333.131040] Key type .llcrypt registered [ 7333.241939] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 7334.305959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7334.346078] alg: No test for adler32 (adler32-zlib) [ 7335.361529] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 7335.562730] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 7337.231187] Key type lgssc registered [ 7338.408388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7346.020475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7346.036939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7347.313610] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7347.336226] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7347.458553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7352.017553] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7361.534669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7361.615601] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7361.640342] Lustre: 254917:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7361.888047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7367.135322] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7374.363439] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7379.509758] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7386.024819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7386.280617] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7386.507385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7393.559195] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7396.554219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7401.251060] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7457.763385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7457.773979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7457.786922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7458.274563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7458.275668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7458.287196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7458.287205] Lustre: Skipped 1 previous similar message [ 7461.019044] Lustre: server umount lustre-OST0000 complete [ 7465.040541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7476.992115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7477.464989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7478.515141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7478.538615] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7483.502416] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7483.707844] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7483.722873] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7490.047807] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7529.953791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7529.967730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7529.992660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7530.000344] Lustre: Skipped 1 previous similar message [ 7533.029368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7533.044806] Lustre: Skipped 1 previous similar message [ 7534.255280] Lustre: server umount lustre-OST0000 complete [ 7538.153538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7538.155450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7538.181806] Lustre: Skipped 2 previous similar messages [ 7543.848423] Lustre: server umount lustre-MDT0000 complete [ 7547.000918] LustreError: 254910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384352 with bad export cookie 11367642898471245790 [ 7547.004624] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7547.013179] LustreError: 254910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7547.367317] Lustre: server umount lustre-MDT0001 complete [ 7556.918344] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 7559.144934] Key type lgssc unregistered [ 7559.433506] LNet: 260345:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7559.450422] LNetError: 260345:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7559.465062] LNet: Removed LNI 192.168.206.128@tcp [ 7560.232159] Key type .llcrypt unregistered [ 7560.234184] Key type ._llcrypt unregistered [ 7582.200512] Key type ._llcrypt registered [ 7582.202351] Key type .llcrypt registered [ 7582.284581] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 7583.203936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7583.375410] alg: No test for adler32 (adler32-zlib) [ 7584.506308] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 7584.748601] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 7586.424653] Key type lgssc registered [ 7587.436639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7593.776800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7593.802400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7595.190818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7599.453160] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7604.712406] LustreError: 261766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7604.733407] LustreError: 261766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7607.660145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7608.068403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7612.571940] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7620.521538] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7626.730787] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7633.577267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7633.838387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7639.560203] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7639.980490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7647.205396] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7672.804372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7672.809274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7672.815348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7672.816394] Lustre: Skipped 1 previous similar message [ 7677.923181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7677.934870] Lustre: Skipped 1 previous similar message [ 7678.560272] Lustre: server umount lustre-OST0000 complete [ 7683.045738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7683.058837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7683.059085] Lustre: Skipped 1 previous similar message [ 7683.072916] Lustre: Skipped 1 previous similar message [ 7688.163486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7688.168753] Lustre: Skipped 2 previous similar messages [ 7688.979089] Lustre: server umount lustre-MDT0000 complete [ 7692.999817] LustreError: 261745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384498 with bad export cookie 12334819843960822772 [ 7693.009408] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7693.010909] LustreError: 261745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7693.462653] Lustre: server umount lustre-MDT0001 complete [ 7703.663432] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 7706.214835] Key type lgssc unregistered [ 7706.516682] LNet: 266070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7706.531589] LNetError: 266070:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7706.557415] LNet: Removed LNI 192.168.206.128@tcp [ 7707.404466] Key type .llcrypt unregistered [ 7707.408119] Key type ._llcrypt unregistered [ 7723.049778] Key type ._llcrypt registered [ 7723.053845] Key type .llcrypt registered [ 7723.154614] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 7724.043866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7724.094725] alg: No test for adler32 (adler32-zlib) [ 7725.137049] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 7725.365899] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 7727.095165] Key type lgssc registered [ 7727.943656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7734.841631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7734.857771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7736.195555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7739.997195] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7745.507496] LustreError: 267294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7745.528864] LustreError: 267294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7747.593309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7747.944861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7752.084438] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7758.592747] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7763.235920] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7768.954820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7769.222777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7774.377201] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7780.465685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7781.278470] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7791.072103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7791.081249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7791.083868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7791.092016] Lustre: Skipped 1 previous similar message [ 7791.100606] Lustre: Skipped 1 previous similar message [ 7796.201027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7796.216086] Lustre: Skipped 1 previous similar message [ 7797.088236] Lustre: server umount lustre-OST0000 complete [ 7801.312905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7801.316061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7801.332843] Lustre: Skipped 1 previous similar message [ 7806.434771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7806.444252] Lustre: Skipped 2 previous similar messages [ 7806.667952] Lustre: server umount lustre-MDT0000 complete [ 7810.326260] LustreError: 267274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384616 with bad export cookie 18291732570151587405 [ 7810.341266] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7810.346827] LustreError: 267274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7810.734214] Lustre: server umount lustre-MDT0001 complete [ 7823.735579] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 7827.566452] Key type lgssc unregistered [ 7827.924605] LNet: 270969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7827.931223] LNetError: 270969:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7827.941899] LNet: Removed LNI 192.168.206.128@tcp [ 7828.962755] Key type .llcrypt unregistered [ 7828.964340] Key type ._llcrypt unregistered [ 7853.787118] Key type ._llcrypt registered [ 7853.793459] Key type .llcrypt registered [ 7853.960206] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 7854.817578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7854.855449] alg: No test for adler32 (adler32-zlib) [ 7855.967313] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 7856.207229] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 7857.911423] Key type lgssc registered [ 7859.131827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7866.262131] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7866.292190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7867.742952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7872.469050] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7878.120178] LustreError: 272385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7878.141425] LustreError: 272385:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7880.945353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7881.266050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7885.506923] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7892.839789] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7898.367625] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7904.241581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7904.498519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7910.033719] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 7912.721240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7918.429396] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7938.531398] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7938.537739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7938.549512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7938.551994] Lustre: Skipped 1 previous similar message [ 7943.651640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7944.192791] Lustre: server umount lustre-OST0000 complete [ 7948.775598] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7948.785513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7948.820258] Lustre: Skipped 1 previous similar message [ 7948.820906] Lustre: Skipped 2 previous similar messages [ 7953.897709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7953.913833] Lustre: Skipped 2 previous similar messages [ 7954.562901] Lustre: server umount lustre-MDT0000 complete [ 7958.578833] LustreError: 272365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384764 with bad export cookie 2923370672445161435 [ 7958.586504] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7958.597787] LustreError: 272365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7959.081810] Lustre: server umount lustre-MDT0001 complete [ 7968.182603] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 7970.877384] Key type lgssc unregistered [ 7971.222102] LNet: 276694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7971.228358] LNetError: 276694:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7971.257786] LNet: Removed LNI 192.168.206.128@tcp [ 7971.963736] Key type .llcrypt unregistered [ 7971.965482] Key type ._llcrypt unregistered [ 7985.535954] Key type ._llcrypt registered [ 7985.538480] Key type .llcrypt registered [ 7985.671410] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 7986.596625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7986.641504] alg: No test for adler32 (adler32-zlib) [ 7987.651230] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 7987.838551] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 7989.471190] Key type lgssc registered [ 7990.405201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7996.884300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7996.906465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7998.257071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8002.383198] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8008.688249] LustreError: 277910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8008.712273] LustreError: 277910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8010.175209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8010.494069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8014.449444] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8021.679857] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8026.927582] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8034.385912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8034.673849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8039.811951] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8043.896414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8045.340508] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8056.289310] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8056.303800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8056.317294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8057.013470] Lustre: server umount lustre-OST0000 complete [ 8061.920596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8061.932990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8061.945295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8064.480346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8064.480904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8064.507874] Lustre: Skipped 1 previous similar message [ 8069.601147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8069.604099] Lustre: Skipped 1 previous similar message [ 8070.717912] Lustre: server umount lustre-MDT0000 complete [ 8074.269541] LustreError: 277877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384880 with bad export cookie 17089769264043777722 [ 8074.276202] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8074.282427] LustreError: 277877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8074.712469] Lustre: server umount lustre-MDT0001 complete [ 8085.886926] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 8088.510084] Key type lgssc unregistered [ 8088.846850] LNet: 281610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8088.860593] LNetError: 281610:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8088.902592] LNet: Removed LNI 192.168.206.128@tcp [ 8089.758583] Key type .llcrypt unregistered [ 8089.760940] Key type ._llcrypt unregistered [ 8101.482121] Key type ._llcrypt registered [ 8101.484034] Key type .llcrypt registered [ 8101.590765] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8116.620990] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 8132.134461] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 8133.375447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8133.633061] alg: No test for adler32 (adler32-zlib) [ 8134.721536] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 8134.932638] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 8136.583217] Key type lgssc registered [ 8137.629616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8171.325540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8171.362921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8172.630567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8172.675792] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8172.762931] Lustre: lustre-MDT0000: new disk, initializing [ 8172.848821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8172.882601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8177.135583] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8186.447901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8186.550854] Lustre: 285167:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8186.599295] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8186.603239] Lustre: Skipped 1 previous similar message [ 8186.664207] Lustre: lustre-MDT0001: new disk, initializing [ 8186.763871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8186.795734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8186.807327] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8191.326711] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8201.457638] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8206.447446] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8213.211055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8213.513141] Lustre: lustre-OST0000: new disk, initializing [ 8213.520677] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8213.534180] Lustre: 287134:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8213.635889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8219.580568] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8220.268593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8220.285547] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8220.428628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8229.091464] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8233.749219] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8234.005528] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8239.230520] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8239.490399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8246.241226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8246.247619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8246.253369] LustreError: Skipped 1 previous similar message [ 8246.255623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8246.268102] Lustre: Skipped 1 previous similar message [ 8247.893693] Lustre: server umount lustre-OST0000 complete [ 8251.362188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8251.383932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8251.390014] Lustre: Skipped 2 previous similar messages [ 8256.482202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8256.492039] Lustre: Skipped 1 previous similar message [ 8257.670083] Lustre: server umount lustre-MDT0000 complete [ 8261.370188] LustreError: 285961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385067 with bad export cookie 4836859969096380692 [ 8261.375524] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8261.397318] LustreError: 285961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8261.796092] Lustre: server umount lustre-MDT0001 complete [ 8269.725597] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:51:14 (1787385074) [ 8275.388984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8276.087924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8280.380480] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8281.569703] LustreError: 289545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8281.578375] LustreError: 289545:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8285.670447] LustreError: 289546:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8288.120429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8288.391379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8291.950519] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8299.554482] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8305.591811] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8312.152633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8312.393061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8317.752263] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8324.346467] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8334.304771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8334.321055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8334.333787] Lustre: Skipped 1 previous similar message [ 8334.337795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8336.356372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8338.619924] Lustre: server umount lustre-OST0000 complete [ 8345.058456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8345.072640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8345.089464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8345.094200] Lustre: Skipped 2 previous similar messages [ 8348.452464] Lustre: server umount lustre-MDT0000 complete [ 8351.712435] LustreError: 291864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8351.730326] LustreError: 291864:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8351.986896] LustreError: 289526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385157 with bad export cookie 4836859969096381588 [ 8351.987665] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8352.003085] LustreError: 289526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8352.424236] Lustre: server umount lustre-MDT0001 complete [ 8364.049795] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 8366.688992] Key type lgssc unregistered [ 8366.985663] LNet: 293082:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8366.994637] LNetError: 293082:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8368.037968] LNet: Removed LNI 192.168.206.128@tcp [ 8368.797763] Key type .llcrypt unregistered [ 8368.800530] Key type ._llcrypt unregistered [ 8396.382328] Key type ._llcrypt registered [ 8396.393884] Key type .llcrypt registered [ 8396.504301] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 8410.392636] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 8411.304228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8411.324501] alg: No test for adler32 (adler32-zlib) [ 8412.329501] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 8412.496645] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 8414.167223] Key type lgssc registered [ 8415.056347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8453.084683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8453.117542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8454.411095] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8454.451145] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8454.522194] Lustre: lustre-MDT0000: new disk, initializing [ 8454.620198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8454.650466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8458.653302] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8468.111440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8468.187646] Lustre: 296906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8468.233655] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8468.237393] Lustre: Skipped 1 previous similar message [ 8468.311822] Lustre: lustre-MDT0001: new disk, initializing [ 8468.406711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8468.430836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8468.438393] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8471.965725] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8481.932955] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8486.613134] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8491.624166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8491.875521] Lustre: lustre-OST2710: new disk, initializing [ 8491.879561] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8491.883682] Lustre: 298877:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8491.939903] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8495.478617] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8499.241196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8499.250202] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8499.332410] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8503.326519] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8509.612654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8509.818555] Lustre: lustre-OST03e8: new disk, initializing [ 8509.821962] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8509.828977] Lustre: 299940:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8509.931693] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8516.585841] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8519.734151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8519.755521] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8519.809548] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8523.205915] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8529.663450] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8529.833124] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8534.316800] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8534.560352] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8538.653664] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8538.923131] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8543.130484] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8543.465429] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8549.612456] Lustre: server umount lustre-MDT0000 complete [ 8550.378312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8550.379892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8550.394777] Lustre: Skipped 1 previous similar message [ 8552.742577] LustreError: 296898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385358 with bad export cookie 14936737170195447584 [ 8552.753473] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8552.754185] LustreError: 296898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8552.997167] Lustre: server umount lustre-MDT0001 complete [ 8566.685567] Lustre: server umount lustre-OST2710 complete [ 8580.078969] Lustre: server umount lustre-OST03e8 complete [ 8596.357184] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 8606.245193] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 8639.108379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8639.284753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8639.304066] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8639.356534] Lustre: lustre-MDT0000: new disk, initializing [ 8639.408780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8639.422785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8643.343196] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8653.074088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8653.254486] Lustre: 305189:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8653.298383] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8653.302394] Lustre: Skipped 1 previous similar message [ 8653.363338] Lustre: lustre-MDT0001: new disk, initializing [ 8653.439555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8653.462812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8653.479775] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8658.022546] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8667.224997] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8672.173479] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8678.048616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8678.269874] Lustre: lustre-OST0000: new disk, initializing [ 8678.273546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8678.278414] Lustre: 307159:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8678.351813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8684.132292] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8684.610277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8684.624912] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8684.803297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8693.070034] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8697.529339] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8697.799734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8702.425782] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8702.678272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8705.512982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8705.519677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8705.523250] LustreError: Skipped 1 previous similar message [ 8705.524389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8705.544475] Lustre: Skipped 3 previous similar messages [ 8705.558928] Lustre: Skipped 1 previous similar message [ 8710.628379] LustreError: 307141:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8710.649729] LustreError: 307141:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8710.792629] Lustre: server umount lustre-OST0000 complete [ 8715.235637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8715.244762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8715.266188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8720.663519] Lustre: server umount lustre-MDT0000 complete [ 8720.865083] LustreError: 305200:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8720.890587] LustreError: 305200:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8724.716332] LustreError: 305182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385530 with bad export cookie 14936737170195449264 [ 8724.721936] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8724.733708] LustreError: 305182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8725.239462] Lustre: server umount lustre-MDT0001 complete [ 8733.570490] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:58:58 (1787385538) [ 8735.195515] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8736.588277] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:59:01 (1787385541) [ 8739.705084] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 8764.269840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8764.573428] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8764.594216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8764.790186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8769.861836] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8778.604977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8778.766284] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8778.802783] Lustre: 311014:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8783.265514] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8805.103177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8805.312484] Lustre: Denying initial registration attempt from nid 192.168.206.128@tcp, specified as failover [ 8805.326276] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8805.332725] LustreError: 312467:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8805.342929] LustreError: 312467:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8805.356179] LustreError: 312467:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8805.414296] Lustre: server umount lustre-OST0000 complete [ 8805.416386] LustreError: 312467:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8814.048530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8814.066130] Lustre: Skipped 1 previous similar message [ 8814.070418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8814.073913] Lustre: Skipped 2 previous similar messages [ 8817.718880] Lustre: server umount lustre-MDT0000 complete [ 8819.169899] LustreError: 311028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8819.183125] LustreError: 311028:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8821.147692] LustreError: 311757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385626 with bad export cookie 14936737170195450146 [ 8821.151894] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8821.168017] LustreError: 311757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8821.377129] Lustre: server umount lustre-MDT0001 complete [ 8833.311556] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 8835.971800] Key type lgssc unregistered [ 8836.191728] LNet: 313591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8836.202076] LNetError: 313591:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8836.213559] LNet: Removed LNI 192.168.206.128@tcp [ 8836.953251] Key type .llcrypt unregistered [ 8836.958220] Key type ._llcrypt unregistered [ 8848.357609] Key type ._llcrypt registered [ 8848.359690] Key type .llcrypt registered [ 8848.480676] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 8849.140697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8849.228692] alg: No test for adler32 (adler32-zlib) [ 8850.238497] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 8850.429559] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 8852.143180] Key type lgssc registered [ 8853.117671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8896.661846] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 8905.550872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8905.575756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8906.870550] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8906.889179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8907.067206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8911.430861] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8920.173129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8920.241942] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8920.273268] Lustre: 316371:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8920.585967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8925.555715] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8933.105810] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8939.064098] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8958.718892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8958.949649] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8959.150912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8964.955508] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 8971.728280] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8977.388983] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8977.401913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8977.421947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8977.891989] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8977.918848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8977.940288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8980.237041] Lustre: server umount lustre-OST0000 complete [ 8987.106371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8987.120405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8987.153677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8989.777778] Lustre: server umount lustre-MDT0000 complete [ 8993.412652] LustreError: 316364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385799 with bad export cookie 16971251582789749795 [ 8993.417943] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8993.427030] LustreError: 316364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8993.682708] Lustre: server umount lustre-MDT0001 complete [ 9003.252723] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 9006.038830] Key type lgssc unregistered [ 9006.355413] LNet: 319942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9006.359690] LNetError: 319942:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9006.376348] LNet: Removed LNI 192.168.206.128@tcp [ 9007.236173] Key type .llcrypt unregistered [ 9007.239946] Key type ._llcrypt unregistered [ 9031.610767] Key type ._llcrypt registered [ 9031.614483] Key type .llcrypt registered [ 9031.747243] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 9032.936307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9032.966642] alg: No test for adler32 (adler32-zlib) [ 9034.082807] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 9034.324251] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 9036.017093] Key type lgssc registered [ 9037.296694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9045.935212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9045.975712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9047.467138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9051.398323] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9057.766962] LustreError: 321361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9057.793629] LustreError: 321361:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9059.326232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9059.919444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9064.415184] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9071.944666] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9077.631661] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9086.902692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9087.404187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9094.916554] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9102.971599] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9115.115059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9115.118270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9115.139405] Lustre: Skipped 2 previous similar messages [ 9115.150305] Lustre: Skipped 1 previous similar message [ 9116.131363] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.28@tcp (stopping) [ 9118.396174] Lustre: server umount lustre-MDT0000 complete [ 9120.227260] LustreError: 321360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9120.271226] LustreError: 321360:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9122.365424] LustreError: 321340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385928 with bad export cookie 15131387142261599539 [ 9122.371901] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9122.373595] LustreError: 321340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9125.344069] LustreError: 323947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9125.347723] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9125.372664] LustreError: 323947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9125.382553] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9127.392323] LustreError: 321357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9127.396328] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9127.411027] LustreError: 321357:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9127.432765] Lustre: Skipped 1 previous similar message [ 9128.767486] Lustre: server umount lustre-MDT0001 complete [ 9135.449507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9144.324751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9147.873448] LustreError: 320668:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f49c8787 x1874209894605952/t0(0) o250->MGC192.168.206.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9148.202253] LustreError: 323193:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9148.291283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9148.360261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9152.896752] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9157.474967] LustreError: 324957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9157.479916] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9157.489950] LustreError: 324957:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9161.754743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9162.294073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9167.217181] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9167.378185] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9175.596772] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9177.202051] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9181.912229] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9183.756626] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9189.345058] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9189.360951] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9189.385635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9192.933316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9198.063984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9198.068541] Lustre: Skipped 3 previous similar messages [ 9203.167125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9203.401301] Lustre: server umount lustre-OST0000 complete [ 9208.291474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9208.294764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9213.940437] Lustre: server umount lustre-MDT0000 complete [ 9217.350901] LustreError: 324150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386023 with bad export cookie 15131387142261607218 [ 9217.354823] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9217.364091] LustreError: 324150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9217.715862] Lustre: server umount lustre-MDT0001 complete [ 9226.177332] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 9229.005674] Key type lgssc unregistered [ 9229.352064] LNet: 327633:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9229.364065] LNetError: 327633:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9229.391260] LNet: Removed LNI 192.168.206.128@tcp [ 9230.283597] Key type .llcrypt unregistered [ 9230.286272] Key type ._llcrypt unregistered [ 9254.523739] Key type ._llcrypt registered [ 9254.527294] Key type .llcrypt registered [ 9254.659844] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 9255.682689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9255.775049] alg: No test for adler32 (adler32-zlib) [ 9256.886934] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 9257.133952] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 9258.824820] Key type lgssc registered [ 9260.042413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9267.110653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9267.134952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9268.564450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9273.372964] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9278.951478] LustreError: 329050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9278.980630] LustreError: 329050:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9279.976251] LustreError: 329049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9282.393353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9282.819416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9287.874365] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9294.858841] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9299.904879] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9303.520893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9303.530676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9303.549605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9304.545487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9304.550031] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9307.694925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9307.702731] Lustre: Skipped 2 previous similar messages [ 9308.614557] Lustre: server umount lustre-MDT0000 complete [ 9312.555099] LustreError: 329030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386118 with bad export cookie 4448821551397293678 [ 9312.567139] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9312.572450] LustreError: 329030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9313.032810] Lustre: server umount lustre-MDT0001 complete [ 9326.504446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9326.853712] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9326.880817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9327.090499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9331.731654] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9340.069233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9340.229947] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9340.284991] Lustre: 331638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9340.695841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9344.910993] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9351.265178] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9356.442215] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9365.552915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9365.745440] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9365.752601] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9365.757975] LustreError: 333522:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9365.762324] LustreError: 333522:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9365.767535] LustreError: 333522:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9365.816896] Lustre: server umount lustre-OST0000 complete [ 9365.820461] LustreError: 333522:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9376.041250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9376.203063] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9376.408631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9382.179766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9383.798452] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9391.706416] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9399.268714] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9399.384447] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9399.439924] Lustre: lustre-OST0001: new disk, initializing [ 9399.443353] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9399.447313] Lustre: 335061:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9399.496490] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9405.148973] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9408.151790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9408.167561] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9408.269747] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9412.451593] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9418.721707] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9418.729538] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9418.734113] LustreError: Skipped 1 previous similar message [ 9418.735056] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9418.746335] Lustre: Skipped 1 previous similar message [ 9421.592840] Lustre: server umount lustre-OST0001 complete [ 9423.844772] LustreError: 335542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9423.863794] LustreError: 335542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9428.450103] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9428.453923] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9428.466576] LustreError: Skipped 1 previous similar message [ 9428.484362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9428.492459] Lustre: Skipped 2 previous similar messages [ 9428.970435] LustreError: 335542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9428.993263] LustreError: 335542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9431.554307] Lustre: server umount lustre-OST0000 complete [ 9438.177302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9438.184981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9438.216707] Lustre: Skipped 1 previous similar message [ 9438.230696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9441.768915] Lustre: server umount lustre-MDT0000 complete [ 9444.323589] LustreError: 332465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9444.335617] LustreError: 332465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9445.833162] LustreError: 331630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386251 with bad export cookie 4448821551397294231 [ 9445.850014] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9445.857461] LustreError: 331630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9446.384423] Lustre: server umount lustre-MDT0001 complete [ 9457.711780] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 9460.677407] Key type lgssc unregistered [ 9461.044644] LNet: 337023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9461.052414] LNetError: 337023:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9461.088674] LNet: Removed LNI 192.168.206.128@tcp [ 9462.045613] Key type .llcrypt unregistered [ 9462.047814] Key type ._llcrypt unregistered [ 9532.248632] Key type ._llcrypt registered [ 9532.251718] Key type .llcrypt registered [ 9532.371597] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 9547.790332] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 9548.965361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9548.981596] alg: No test for adler32 (adler32-zlib) [ 9550.038868] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [ 9550.324593] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 9551.991163] Key type lgssc registered [ 9553.207541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9588.201645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9588.220643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9589.410765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9589.431785] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9589.489380] Lustre: lustre-MDT0000: new disk, initializing [ 9589.559943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9589.589327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9593.383963] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9603.652911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9603.792214] Lustre: 341955:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9603.913589] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9603.920384] Lustre: Skipped 1 previous similar message [ 9604.005107] Lustre: lustre-MDT0001: new disk, initializing [ 9604.114504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9604.203304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9604.215563] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9608.298511] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9616.264666] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9620.481518] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9625.343752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9625.595337] Lustre: lustre-OST0000: new disk, initializing [ 9625.598778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9625.603375] Lustre: 343925:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9625.659188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9630.853170] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9634.859875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9634.880715] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9634.990810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9638.858972] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9642.721921] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9642.988846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9646.834869] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9647.009741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9650.659997] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9650.662465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9650.664064] LustreError: Skipped 1 previous similar message [ 9650.665058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9650.675726] Lustre: Skipped 1 previous similar message [ 9650.690122] Lustre: Skipped 1 previous similar message [ 9654.496660] Lustre: server umount lustre-OST0000 complete [ 9660.386439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9660.396528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9660.411413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9662.861410] Lustre: server umount lustre-MDT0000 complete [ 9666.097557] LustreError: 343924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386471 with bad export cookie 13415861247894543536 [ 9666.099273] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9666.107952] LustreError: 343924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9666.370629] Lustre: server umount lustre-MDT0001 complete [ 9673.279883] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 04:14:37 (1787386477) [ 9679.501746] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 04:14:44 (1787386484) [ 9686.627130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9686.995420] LustreError: 346678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9687.011748] LustreError: 346678:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9687.065358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9690.585569] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9692.128240] LustreError: 346679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9692.142518] LustreError: 346679:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9696.227318] LustreError: 346678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9697.217910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9697.519072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9701.261814] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9707.328258] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9711.669315] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9716.962148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9717.212820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9723.367983] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9729.771622] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9742.317138] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9744.081418] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9745.461358] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9747.123483] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9748.932884] Lustre: Failing over lustre-MDT0000 [ 9748.964427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9748.969663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9748.977819] Lustre: Skipped 1 previous similar message [ 9748.983994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9749.001360] Lustre: Skipped 1 previous similar message [ 9749.366314] Lustre: server umount lustre-MDT0000 complete [ 9752.012875] LustreError: 346673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9752.037317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9752.058184] Lustre: Skipped 1 previous similar message [ 9757.105141] LustreError: 349082:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9757.124018] LustreError: 349082:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9767.348630] LustreError: 346673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9767.379063] LustreError: 346673:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 9767.433247] Lustre: 340179:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787386557/real 1787386557] req@00000000bf9c7700 x1874210435695104/t0(0) o400->MGC192.168.206.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787386573 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9767.467152] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9767.522534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9777.633577] LustreError: 340178:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000087e36138 x1874210435699840/t0(0) o250->MGC192.168.206.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9777.999486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9778.048859] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9782.078224] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9782.720326] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9783.273127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9783.305753] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9783.353205] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9790.565168] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9792.031061] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9795.620195] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9797.462149] Lustre: Failing over lustre-MDT0000 [ 9797.684371] Lustre: server umount lustre-MDT0000 complete [ 9798.116980] LustreError: 346673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9798.132083] LustreError: 346673:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 9798.623974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9798.629926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9798.643547] Lustre: Skipped 2 previous similar messages [ 9815.008963] Lustre: 340182:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787386604/real 1787386604] req@0000000091ae0f92 x1874210435714176/t0(0) o400->MGC192.168.206.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787386620 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9815.059963] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9815.560145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9824.225302] LustreError: 340178:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f35b9389 x1874210435719552/t0(0) o250->MGC192.168.206.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9824.487048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9824.538929] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9828.771462] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9828.804269] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9829.884553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9829.898908] Lustre: Skipped 2 previous similar messages [ 9829.933695] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9829.974533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9837.985718] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9839.676553] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9845.219563] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9845.221806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9845.229518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9845.243343] Lustre: Skipped 1 previous similar message [ 9845.255054] Lustre: Skipped 2 previous similar messages [ 9850.342733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9850.351529] Lustre: Skipped 2 previous similar messages [ 9851.138591] Lustre: server umount lustre-MDT0000 complete [ 9854.379181] LustreError: 348519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386660 with bad export cookie 13415861247894546252 [ 9854.389368] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9854.390892] LustreError: 348519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9854.846680] Lustre: server umount lustre-MDT0001 complete [ 9867.588733] Lustre: server umount lustre-OST0000 complete [ 9883.805372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9884.303466] LustreError: 353041:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9884.327718] LustreError: 353041:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 9889.116350] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9897.652719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9898.011358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9898.017991] Lustre: Skipped 1 previous similar message [ 9902.129693] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9910.029937] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9915.893799] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9923.754287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9929.399996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9931.118963] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 9939.320574] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9950.186427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9950.200758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9950.221211] Lustre: Skipped 1 previous similar message [ 9950.232961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9956.055331] Lustre: server umount lustre-OST0000 complete [ 9966.372515] Lustre: server umount lustre-MDT0000 complete [ 9968.614976] LustreError: 353041:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9968.642524] LustreError: 353041:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9970.134087] LustreError: 353022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386775 with bad export cookie 13415861247894546994 [ 9970.135957] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9970.159492] LustreError: 353022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9970.543943] Lustre: server umount lustre-MDT0001 complete [ 9981.900229] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 9985.077413] Key type lgssc unregistered [ 9985.472416] LNet: 356574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9985.494517] LNetError: 356574:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9985.534751] LNet: Removed LNI 192.168.206.128@tcp [ 9986.616624] Key type .llcrypt unregistered [ 9986.620077] Key type ._llcrypt unregistered [10013.044472] Key type ._llcrypt registered [10013.047703] Key type .llcrypt registered [10013.215288] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [10014.411224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10014.476898] alg: No test for adler32 (adler32-zlib) [10015.774875] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [10016.099883] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [10017.840853] Key type lgssc registered [10019.253610] Lustre: Echo OBD driver; http://www.lustre.org/ [10027.293856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10027.343460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10028.929981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10033.523899] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10038.255436] LustreError: 358090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10038.283854] LustreError: 358090:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10042.471332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10042.811643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10047.757348] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10055.502605] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10061.148862] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10067.585957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10068.043605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10074.187478] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10074.222522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10081.601386] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10097.408047] Lustre: server umount lustre-MDT0000 complete [10098.153295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10098.157399] LustreError: 358090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10098.170114] Lustre: Skipped 2 previous similar messages [10098.194875] LustreError: 358090:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10101.280969] LustreError: 358071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386907 with bad export cookie 3466333469062715919 [10101.289410] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10101.300234] LustreError: 358071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10101.688346] Lustre: server umount lustre-MDT0001 complete [10115.810126] Lustre: server umount lustre-OST0000 complete [10128.604196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10128.995328] LustreError: 361821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10129.071368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10134.286688] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10134.502662] LustreError: 361822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10134.526143] LustreError: 361822:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10138.595315] LustreError: 361821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10143.422331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10143.743574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10148.304611] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10155.261867] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10161.692591] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10169.213840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10169.783183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10173.890123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10177.644718] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10185.034963] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10199.523285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10199.527551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10199.534071] Lustre: Skipped 2 previous similar messages [10199.547580] Lustre: Skipped 2 previous similar messages [10203.297486] Lustre: server umount lustre-MDT0000 complete [10204.644420] LustreError: 362549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10204.676590] LustreError: 362549:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10207.768584] LustreError: 361803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387013 with bad export cookie 3466333469062717032 [10207.774019] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10207.778588] LustreError: 361803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10208.028955] Lustre: server umount lustre-MDT0001 complete [10223.053432] Lustre: server umount lustre-OST0000 complete [10238.729703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10239.323691] LustreError: 365730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10239.403597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10245.042595] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10256.348286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10262.549820] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10270.571215] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10276.791821] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10284.583632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10285.115036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10285.121950] Lustre: Skipped 1 previous similar message [10291.490886] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10299.028660] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10309.712116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10316.770032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10316.775882] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10316.798217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10319.851060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10319.853039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10321.744799] Lustre: server umount lustre-OST0000 complete [10328.543956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10328.555196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10328.574485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10328.587991] Lustre: Skipped 1 previous similar message [10331.721392] Lustre: server umount lustre-MDT0000 complete [10335.200743] LustreError: 365730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10335.226876] LustreError: 365730:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10335.311213] LustreError: 365712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387141 with bad export cookie 3466333469062718054 [10335.328154] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10335.329521] LustreError: 365712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10335.843995] Lustre: server umount lustre-MDT0001 complete [10348.156801] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [10352.005438] Key type lgssc unregistered [10352.386741] LNet: 369288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10352.410302] LNetError: 369288:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10352.454715] LNet: Removed LNI 192.168.206.128@tcp [10353.598492] Key type .llcrypt unregistered [10353.601486] Key type ._llcrypt unregistered [10377.256939] Key type ._llcrypt registered [10377.260605] Key type .llcrypt registered [10377.395329] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [10378.577294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10378.592867] alg: No test for adler32 (adler32-zlib) [10379.790922] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [10380.049634] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [10381.719551] Key type lgssc registered [10382.568890] Lustre: Echo OBD driver; http://www.lustre.org/ [10391.193832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10391.232026] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10391.247947] LustreError: 370734:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10391.277913] LustreError: 370734:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10391.285315] LustreError: 370734:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10391.293914] LustreError: 370734:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10391.314404] LustreError: 370734:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10399.151695] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10399.156677] LustreError: 371131:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10399.171549] LustreError: 371131:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10399.176156] LustreError: 371131:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10399.181552] LustreError: 371131:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10399.188692] LustreError: 371131:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10419.458608] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [10422.873374] Key type lgssc unregistered [10423.345915] LNet: 372134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10423.357648] LNetError: 372134:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10423.373971] LNet: Removed LNI 192.168.206.128@tcp [10424.363340] Key type .llcrypt unregistered [10424.366868] Key type ._llcrypt unregistered [10442.099504] Key type ._llcrypt registered [10442.104719] Key type .llcrypt registered [10442.269392] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [10459.082922] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [10460.559955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10460.587569] alg: No test for adler32 (adler32-zlib) [10461.651854] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [10462.082346] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [10463.786848] Key type lgssc registered [10465.264744] Lustre: Echo OBD driver; http://www.lustre.org/ [10499.344355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10499.366022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10500.693079] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10500.721125] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10500.845139] Lustre: lustre-MDT0000: new disk, initializing [10500.929030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10500.945638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10505.088364] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10514.827146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10515.012909] Lustre: 375294:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10515.086456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10515.094070] Lustre: Skipped 1 previous similar message [10515.176669] Lustre: lustre-MDT0001: new disk, initializing [10515.312424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10515.374625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10515.381638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10519.505385] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10528.412800] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10533.728665] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10539.586607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10539.799194] Lustre: lustre-OST0000: new disk, initializing [10539.803964] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10539.809394] Lustre: 377262:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10539.855797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10545.248916] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10549.857610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10549.867966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10549.933667] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10554.914461] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10559.608674] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10560.013453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10566.430445] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10566.795497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10570.728994] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10570.730983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10570.744402] LustreError: Skipped 1 previous similar message [10570.746398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10570.824531] Lustre: Skipped 1 previous similar message [10574.600906] Lustre: server umount lustre-OST0000 complete [10580.971516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10580.973945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10580.986225] Lustre: Skipped 1 previous similar message [10581.002885] Lustre: Skipped 2 previous similar messages [10586.083363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10586.091075] Lustre: Skipped 1 previous similar message [10591.201529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10591.210429] Lustre: Skipped 1 previous similar message [10592.223224] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10592.430974] Lustre: server umount lustre-MDT0000 complete [10596.111327] LustreError: 375285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387401 with bad export cookie 14932240284666276789 [10596.114157] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10596.120966] LustreError: 375285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10596.592774] Lustre: server umount lustre-MDT0001 complete [10604.403802] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10605.988971] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 04:30:10 (1787387410) [10612.116132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10612.600640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10616.525740] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10617.827391] LustreError: 379723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10617.852021] LustreError: 379723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10622.949430] LustreError: 379722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10625.108708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10625.356650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10629.856170] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10636.992984] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10643.200964] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10649.832737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10650.283045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10656.580879] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10663.900123] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10670.320984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10670.507283] Lustre: lustre-OST0001: new disk, initializing [10670.513683] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10670.520224] Lustre: 382556:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10670.612922] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10676.178526] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10680.473827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10680.496432] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10680.609537] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10684.736333] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10691.042857] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10691.050831] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10691.055059] LustreError: Skipped 1 previous similar message [10691.056217] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10691.075783] Lustre: Skipped 1 previous similar message [10694.526809] Lustre: server umount lustre-OST0001 complete [10696.167239] LustreError: 381543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10696.188486] LustreError: 381543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10701.291832] LustreError: 381543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10701.314687] LustreError: 381543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10706.425880] LustreError: 383086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10706.454919] LustreError: 383086:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10711.528833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10711.555775] Lustre: Skipped 1 previous similar message [10711.556173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10711.570037] Lustre: Skipped 3 previous similar messages [10716.645624] LustreError: 383086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10716.660214] LustreError: 383086:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10724.831171] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10725.084528] Lustre: server umount lustre-OST0000 complete [10732.002237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10732.029322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10732.045613] Lustre: Skipped 5 previous similar messages [10735.831229] Lustre: server umount lustre-MDT0000 complete [10739.604895] LustreError: 379704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387545 with bad export cookie 14932240284666277671 [10739.611428] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10739.618041] LustreError: 379704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10745.950141] Lustre: server umount lustre-MDT0001 complete [10758.217984] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [10761.398359] Key type lgssc unregistered [10761.785821] LNet: 384546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10761.793763] LNetError: 384546:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10761.810214] LNet: Removed LNI 192.168.206.128@tcp [10762.986234] Key type .llcrypt unregistered [10762.987700] Key type ._llcrypt unregistered [10816.524284] Key type ._llcrypt registered [10816.526825] Key type .llcrypt registered [10816.619807] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [10817.465293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10817.528085] alg: No test for adler32 (adler32-zlib) [10818.556178] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [10818.741412] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [10820.418887] Key type lgssc registered [10821.509620] Lustre: Echo OBD driver; http://www.lustre.org/ [10828.762884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10828.776349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10830.038533] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10830.063495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10830.275411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10834.683869] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10842.155759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10842.304029] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10842.323795] Lustre: 387348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10842.546728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10846.410665] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10853.474859] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10859.372893] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10863.073371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10863.078685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10863.082560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10863.089847] Lustre: Skipped 1 previous similar message [10867.800117] Lustre: server umount lustre-MDT0000 complete [10871.243641] LustreError: 387340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387677 with bad export cookie 9282252253002340895 [10871.253967] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10871.503990] Lustre: server umount lustre-MDT0001 complete [10880.628921] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 04:34:45 (1787387685) [10887.290324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10887.881439] LustreError: 389861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10887.902157] LustreError: 389861:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10887.979878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10893.291024] LustreError: 389862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10893.771513] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10898.401439] LustreError: 389861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10902.083348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10902.370313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10905.804244] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10912.354808] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10917.433757] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10923.191060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10923.318805] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10923.463707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10929.120182] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [10935.842906] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10943.960269] Lustre: Permanently deactivating lustre-OST0000 [10943.997573] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10950.212181] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10951.556438] LustreError: 392798:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10951.569463] LustreError: 392798:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [10951.578365] LustreError: 392798:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10955.331338] LustreError: 392846:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [10955.345345] LustreError: 392846:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10963.473207] Lustre: server umount lustre-OST0000 complete [10966.572850] LustreError: 393097:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [10966.577950] LustreError: 393097:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10969.058889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10969.068864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10969.090510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10969.093320] Lustre: Skipped 1 previous similar message [10972.641763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10972.643128] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10977.760667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10977.785110] Lustre: Skipped 2 previous similar messages [10982.886740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10982.894626] Lustre: Skipped 1 previous similar message [10983.392483] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10983.589396] Lustre: server umount lustre-MDT0000 complete [10986.870960] LustreError: 389841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387792 with bad export cookie 9282252253002341476 [10986.883810] LustreError: 389841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10986.883861] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10997.476322] Lustre: server umount lustre-MDT0001 complete [11005.802680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11010.346684] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11013.386531] LustreError: 394257:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11019.267199] Lustre: 394596:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.128@tcp [11021.404747] Lustre: server umount MGS complete [11032.795331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11033.238042] LustreError: 395321:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11033.249836] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11033.259545] Lustre: Skipped 1 previous similar message [11033.295738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11037.584125] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11038.696134] LustreError: 395322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11044.991960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11045.232439] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11049.464895] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11055.732969] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11060.926735] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11062.636106] Lustre: Permanently reactivating lustre-OST0000 [11062.648867] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11062.654749] Lustre: Skipped 2 previous similar messages [11088.863261] LustreError: 397057:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11096.901764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11097.223305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11097.227261] Lustre: Skipped 1 previous similar message [11102.297417] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11109.034369] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11113.956640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11113.958068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11113.969035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11113.977228] LustreError: Skipped 1 previous similar message [11115.442635] Lustre: server umount lustre-OST0000 complete [11119.073160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11119.096868] Lustre: Skipped 1 previous similar message [11125.140120] Lustre: server umount lustre-MDT0000 complete [11128.487106] LustreError: 396894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387934 with bad export cookie 9282252253002342526 [11128.492736] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11128.504545] LustreError: 396894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11128.798590] Lustre: server umount lustre-MDT0001 complete [11139.330505] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [11141.638704] Key type lgssc unregistered [11141.903941] LNet: 399685:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11141.913331] LNetError: 399685:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11142.974685] LNet: Removed LNI 192.168.206.128@tcp [11143.920294] Key type .llcrypt unregistered [11143.923705] Key type ._llcrypt unregistered [11160.989246] Key type ._llcrypt registered [11160.997029] Key type .llcrypt registered [11161.105324] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [11174.379211] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [11175.427936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11175.461824] alg: No test for adler32 (adler32-zlib) [11176.507919] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [11176.747162] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [11178.459428] Key type lgssc registered [11179.404629] Lustre: Echo OBD driver; http://www.lustre.org/ [11212.245505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11212.279626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11213.605053] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11213.635714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11213.731636] Lustre: lustre-MDT0000: new disk, initializing [11213.825342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11213.864732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11218.615247] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11228.382357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11228.452061] Lustre: 402842:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11228.493959] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11228.497139] Lustre: Skipped 1 previous similar message [11228.568833] Lustre: lustre-MDT0001: new disk, initializing [11228.668885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11228.707351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11228.726179] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11233.384990] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11242.357204] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11249.643564] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11258.740598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11259.398435] Lustre: lustre-OST0000: new disk, initializing [11259.408946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11259.447824] Lustre: 404811:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11259.625265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11267.810775] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11269.179990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11269.201984] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11269.335762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11277.170496] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11281.006432] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11281.216311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11286.316761] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11286.601936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11290.088169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11290.088223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11290.098431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11290.113352] LustreError: Skipped 1 previous similar message [11294.593585] Lustre: server umount lustre-OST0000 complete [11300.321832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11300.336711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11300.357022] Lustre: Skipped 2 previous similar messages [11300.366833] Lustre: Skipped 1 previous similar message [11304.374313] Lustre: server umount lustre-MDT0000 complete [11308.249571] LustreError: 402835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388114 with bad export cookie 1061496534937922258 [11308.253330] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11308.258013] LustreError: 402835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11308.653140] Lustre: server umount lustre-MDT0001 complete [11317.900972] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11319.859633] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 04:42:04 (1787388124) [11327.353989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11327.823438] LustreError: 407271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11327.843481] LustreError: 407271:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11327.890793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11332.729950] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11333.088952] LustreError: 407272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11338.227538] LustreError: 407271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11340.767810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11341.221476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11345.919366] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11353.068946] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11358.779244] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11364.998632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11365.398834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11371.164940] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11378.784401] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11381.084597] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11383.684519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11391.968670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11391.977492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11391.990405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11391.995257] Lustre: Skipped 1 previous similar message [11394.017849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11394.026074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11394.039535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11395.732698] Lustre: server umount lustre-OST0000 complete [11399.142738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11399.146438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11399.156232] Lustre: Skipped 1 previous similar message [11399.157798] Lustre: Skipped 1 previous similar message [11405.577230] Lustre: server umount lustre-MDT0000 complete [11409.382151] LustreError: 407271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11409.393977] LustreError: 407271:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11410.572220] LustreError: 407252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388216 with bad export cookie 1061496534937923140 [11410.574950] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11410.594361] LustreError: 407252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11411.079149] Lustre: server umount lustre-MDT0001 complete [11422.889649] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [11425.834854] Key type lgssc unregistered [11426.170522] LNet: 410998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11426.182448] LNetError: 410998:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11426.204164] LNet: Removed LNI 192.168.206.128@tcp [11427.087853] Key type .llcrypt unregistered [11427.090528] Key type ._llcrypt unregistered [11451.581681] Key type ._llcrypt registered [11451.586399] Key type .llcrypt registered [11451.729193] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [11452.075562] Key type .llcrypt unregistered [11452.077918] Key type ._llcrypt unregistered [11470.513621] Key type ._llcrypt registered [11470.517862] Key type .llcrypt registered [11470.625349] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [11472.086198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11472.142141] alg: No test for adler32 (adler32-zlib) [11473.212575] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [11473.485519] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [11475.207157] Key type lgssc registered [11476.603880] Lustre: Echo OBD driver; http://www.lustre.org/ [11484.224593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11484.261828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11485.786382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11490.025565] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11495.910679] LustreError: 413213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11495.968329] LustreError: 413213:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11498.461447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11498.810447] LustreError: 413214:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11498.834475] LustreError: 413214:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11498.847143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11504.098678] LustreError: 413213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11504.133127] LustreError: 413213:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11504.514741] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11509.217698] LustreError: 413214:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11509.243706] LustreError: 413214:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11510.260610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11511.642436] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11515.374755] LustreError: 413213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11515.391645] LustreError: 413213:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11518.396990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11518.856041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11524.193628] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11534.305428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11534.308985] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11534.320977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11534.338894] Lustre: Skipped 1 previous similar message [11539.434787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11539.440307] Lustre: Skipped 1 previous similar message [11540.273655] Lustre: server umount lustre-OST0000 complete [11544.545516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11544.549287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11544.571533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11544.575838] Lustre: Skipped 1 previous similar message [11549.665576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11549.675839] Lustre: Skipped 1 previous similar message [11550.401301] Lustre: server umount lustre-MDT0000 complete [11554.140989] LustreError: 413194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388359 with bad export cookie 11905060516993612530 [11554.146389] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11554.157143] LustreError: 413194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11554.622623] Lustre: server umount lustre-MDT0001 complete [11566.171137] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [11569.537520] Key type lgssc unregistered [11569.811576] LNet: 416362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11569.820813] LNetError: 416362:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11569.834746] LNet: Removed LNI 192.168.206.128@tcp [11570.569879] Key type .llcrypt unregistered [11570.575255] Key type ._llcrypt unregistered [11594.272598] Key type ._llcrypt registered [11594.281748] Key type .llcrypt registered [11594.391606] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [11595.541209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11595.565115] alg: No test for adler32 (adler32-zlib) [11596.794651] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [11597.138791] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [11598.855343] Key type lgssc registered [11600.230720] Lustre: Echo OBD driver; http://www.lustre.org/ [11608.769976] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11608.806648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11624.415430] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11629.535829] LustreError: 417751:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.128@tcp: failed processing log, type 4: rc = -110 [11656.223713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11661.752565] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11668.714511] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11675.195553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11675.815816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11675.891424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11680.532335] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11681.248288] LustreError: 417777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11681.280282] LustreError: 417777:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11686.372017] LustreError: 417776:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11689.266390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11689.839237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11695.099876] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11703.044784] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11708.736707] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11717.623595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11717.640054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11717.669619] Lustre: Skipped 1 previous similar message [11722.739757] LustreError: 417767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11722.878766] Lustre: server umount lustre-OST0000 complete [11727.841292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11727.865368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11727.876051] Lustre: Skipped 1 previous similar message [11727.890824] Lustre: Skipped 2 previous similar messages [11732.960789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11732.968477] Lustre: Skipped 1 previous similar message [11733.401566] Lustre: server umount lustre-MDT0000 complete [11736.939635] LustreError: 417758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388542 with bad export cookie 309368672437964455 [11736.947932] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11736.954897] LustreError: 417758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11737.297115] Lustre: server umount lustre-MDT0001 complete [11749.300724] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [11752.574436] Key type lgssc unregistered [11753.007393] LNet: 421322:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11753.021720] LNetError: 421322:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11753.049030] LNet: Removed LNI 192.168.206.128@tcp [11754.040458] Key type .llcrypt unregistered [11754.046466] Key type ._llcrypt unregistered [11778.071795] Key type ._llcrypt registered [11778.076863] Key type .llcrypt registered [11778.189685] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [11779.635037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11779.658252] alg: No test for adler32 (adler32-zlib) [11780.716600] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [11781.019720] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [11782.751160] Key type lgssc registered [11783.981823] Lustre: Echo OBD driver; http://www.lustre.org/ [11791.203703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11791.228488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11792.696348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11796.774876] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11803.114730] LustreError: 422739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11803.150574] LustreError: 422739:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11805.282592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11805.629778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11810.162026] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11817.373097] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11822.315595] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11828.282307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11828.567329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11834.654430] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11837.807390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11840.532556] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11846.633328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11846.646941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11846.660480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11848.137864] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.28@tcp (stopping) [11848.177577] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11848.197539] Lustre: Skipped 1 previous similar message [11851.278557] Lustre: server umount lustre-MDT0000 complete [11853.283116] LustreError: 422739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11853.295345] LustreError: 422739:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11853.586066] LustreError: 422722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.28@tcp arrived at 1787388659 with bad export cookie 14232148567522444748 [11855.396152] LustreError: 422721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388661 with bad export cookie 14232148567522443852 [11855.404770] LustreError: 422721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11855.407248] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11856.352379] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11856.365641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11856.389552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11856.402770] Lustre: Skipped 3 previous similar messages [11858.406084] LustreError: 422740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11858.411914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11858.430379] LustreError: 422740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11860.449786] LustreError: 422739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11860.461169] LustreError: 422739:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11861.658782] Lustre: server umount lustre-OST0000 complete [11865.572098] LustreError: 422740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11865.596327] LustreError: 422740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11867.647733] Lustre: server umount lustre-MDT0001 complete [11877.855410] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [11880.796240] Key type lgssc unregistered [11881.144710] LNet: 426422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11881.156057] LNetError: 426422:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11881.190688] LNet: Removed LNI 192.168.206.128@tcp [11882.017844] Key type .llcrypt unregistered [11882.020833] Key type ._llcrypt unregistered [11904.200906] Key type ._llcrypt registered [11904.206100] Key type .llcrypt registered [11904.330225] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [11905.468176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11905.601432] alg: No test for adler32 (adler32-zlib) [11906.649879] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [11906.887431] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [11908.527225] Key type lgssc registered [11909.717886] Lustre: Echo OBD driver; http://www.lustre.org/ [11916.960801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11916.978271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11918.467197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11922.786509] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11929.062408] LustreError: 427840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11929.079089] LustreError: 427840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11930.998183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11931.280977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11934.880394] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11940.592392] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11945.473147] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11951.333772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11951.636376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11956.939700] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [11962.782977] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11963.959323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11972.581523] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11972.598493] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11972.609868] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11973.584867] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.28@tcp (stopping) [11974.129518] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11976.443341] Lustre: server umount lustre-MDT0001 complete [11979.233097] LustreError: 427839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11979.260306] LustreError: 427839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11984.353277] LustreError: 427840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11984.358368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11984.358618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11984.449441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11984.462115] Lustre: Skipped 2 previous similar messages [11988.580130] Lustre: server umount lustre-OST0000 complete [11989.476293] LustreError: 427839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11992.038771] Lustre: server umount lustre-MDT0000 complete [12003.439536] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [12005.771783] Key type lgssc unregistered [12006.016443] LNet: 431524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12006.028244] LNetError: 431524:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12006.041222] LNet: Removed LNI 192.168.206.128@tcp [12006.654772] Key type .llcrypt unregistered [12006.656404] Key type ._llcrypt unregistered [12026.551291] Key type ._llcrypt registered [12026.554894] Key type .llcrypt registered [12026.647613] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [12038.663983] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [12039.861355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12039.877804] alg: No test for adler32 (adler32-zlib) [12040.925993] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [12041.170710] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [12042.831162] Key type lgssc registered [12044.031142] Lustre: Echo OBD driver; http://www.lustre.org/ [12079.194892] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [12092.622830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12092.655491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12093.914879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12093.955694] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12094.096050] Lustre: lustre-MDT0000: new disk, initializing [12094.234209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12094.271158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12098.849903] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [12112.879940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12113.130607] Lustre: lustre-OST0000: new disk, initializing [12113.136609] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12113.148622] Lustre: Skipped 1 previous similar message [12113.156928] Lustre: 436539:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12113.245700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12118.118296] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [12119.140423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12119.147930] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12119.185164] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12125.224603] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12137.245639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12137.408365] Lustre: 435518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12137.492968] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12137.586736] Lustre: lustre-MDT0001: new disk, initializing [12137.719555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12137.771093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12137.786317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12142.164804] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [12150.006859] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12150.216346] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12153.993458] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12154.233433] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12159.966486] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.28@tcp (stopping) [12160.999145] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12161.001308] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12161.004234] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12161.009871] Lustre: Skipped 1 previous similar message [12161.027165] Lustre: Skipped 1 previous similar message [12164.624937] Lustre: server umount lustre-MDT0001 complete [12171.247119] LustreError: 435527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12171.261341] LustreError: 435527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [12176.353753] LustreError: 435531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12176.365307] LustreError: 435531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12180.452603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12180.461394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12180.476212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12181.477351] LustreError: 435532:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12185.871229] Lustre: server umount lustre-OST0000 complete [12186.603417] LustreError: 435531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12195.551850] Lustre: server umount lustre-MDT0000 complete [12208.891286] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [12211.474536] Key type lgssc unregistered [12211.791729] LNet: 440284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12211.800229] LNetError: 440284:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12211.814731] LNet: Removed LNI 192.168.206.128@tcp [12212.667780] Key type .llcrypt unregistered [12212.671214] Key type ._llcrypt unregistered [12249.729662] Key type ._llcrypt registered [12249.731557] Key type .llcrypt registered [12249.865784] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [12250.140243] Key type .llcrypt unregistered [12250.148494] Key type ._llcrypt unregistered [12265.890431] Key type ._llcrypt registered [12265.898163] Key type .llcrypt registered [12266.022670] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [12267.045932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12267.081111] alg: No test for adler32 (adler32-zlib) [12268.111722] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [12268.343312] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [12270.041291] Key type lgssc registered [12271.165843] Lustre: Echo OBD driver; http://www.lustre.org/ [12294.630898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12294.668084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12296.014220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12296.046687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12296.144677] Lustre: lustre-MDT0000: new disk, initializing [12296.230542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12296.257697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12300.173593] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [12309.938481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12310.020637] Lustre: 443659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12310.070179] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12310.079069] Lustre: Skipped 1 previous similar message [12310.156756] Lustre: lustre-MDT0001: new disk, initializing [12310.261105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12310.297368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12310.312228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12314.848878] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [12323.787199] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12328.826189] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12334.444715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12334.804932] Lustre: lustre-OST0000: new disk, initializing [12334.808405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12334.814122] Lustre: 445630:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12334.882619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12340.029495] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [12344.919383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12344.933506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12345.029320] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12348.168422] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12355.553709] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12355.562259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12355.566917] Lustre: Skipped 2 previous similar messages [12355.581750] Lustre: Skipped 2 previous similar messages [12360.184593] Lustre: server umount lustre-MDT0000 complete [12363.388817] LustreError: 443654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389169 with bad export cookie 3350260249630003178 [12363.394270] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12363.399103] LustreError: 443654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12363.675541] Lustre: server umount lustre-MDT0001 complete [12377.175946] Lustre: server umount lustre-OST0000 complete [12385.618218] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 04:59:50 (1787389190) [12391.751884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12392.171308] LustreError: 447658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12392.192796] LustreError: 447658:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12392.236807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12395.950412] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [12397.542171] LustreError: 447659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12401.636200] LustreError: 447658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12404.528961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12404.803795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12409.396664] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [12417.318147] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12422.501941] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12441.959435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12442.226428] Lustre: Found index 0 for lustre-OST0000, updating log [12442.347679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12448.414235] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [12455.096619] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12462.845130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12473.319289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12473.337582] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12473.340422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12473.355157] Lustre: Skipped 1 previous similar message [12474.406399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12474.416036] Lustre: Skipped 1 previous similar message [12476.803321] Lustre: server umount lustre-OST0000 complete [12491.744129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12491.751092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12491.761490] Lustre: Skipped 1 previous similar message [12491.768963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12495.840288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12495.855215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12497.746163] Lustre: server umount lustre-MDT0000 complete [12500.581317] LustreError: 447640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389306 with bad export cookie 3350260249630004564 [12500.585664] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12500.597933] LustreError: 447640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12500.853937] Lustre: server umount lustre-MDT0001 complete [12505.951144] LNet: 1 local NIs in recovery (showing 1): 192.168.206.128@tcp [12508.233651] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 05:01:53 (1787389313) [13118.945551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13119.326130] LustreError: 454102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13119.388551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13122.875773] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [13124.580613] LustreError: 454103:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13129.700164] LustreError: 454102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13131.212919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13131.570225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13135.840668] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [13142.710591] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13147.911971] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13167.081126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13167.425318] Lustre: Found index 0 for lustre-OST0000, updating log [13167.663223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13172.578162] LNet: 1 local NIs in recovery (showing 1): 192.168.206.128@tcp [13172.726765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [13174.135036] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [13180.766387] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13182.944269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13182.955189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13182.963809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13182.968794] Lustre: Skipped 1 previous similar message [13188.601560] Lustre: server umount lustre-OST0000 complete [13192.419364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13192.435821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13192.442167] Lustre: Skipped 1 previous similar message [13194.467765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13194.473120] Lustre: Skipped 3 previous similar messages [13197.618348] Lustre: server umount lustre-MDT0000 complete [13198.564239] LustreError: 454098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13198.573084] LustreError: 454098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13200.463661] LustreError: 454082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390006 with bad export cookie 3350260249630005656 [13200.470884] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13200.479328] LustreError: 454082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13200.709320] Lustre: server umount lustre-MDT0001 complete [13228.259202] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [13232.991300] LNet: 1 local NIs in recovery (showing 1): 192.168.206.128@tcp [13237.419360] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [13271.259461] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 05:14:35 (1787390075) [13287.895205] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [13290.863329] Key type lgssc unregistered [13291.184337] LNet: 460713:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13291.197573] LNetError: 460713:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13420.270036] LNet: Removed LNI 192.168.206.128@tcp [13421.242205] Key type .llcrypt unregistered [13421.245134] Key type ._llcrypt unregistered [13438.946787] Key type ._llcrypt registered [13438.949970] Key type .llcrypt registered [13439.066898] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [13440.071982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13440.086292] alg: No test for adler32 (adler32-zlib) [13441.265838] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [13441.549221] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [13443.215255] Key type lgssc registered [13444.636592] Lustre: Echo OBD driver; http://www.lustre.org/ [13449.858993] LNet: Added LNI 192.168.206.128@tcp100 [8/256/0/180] [13450.674210] LNet: Added LNI 192.168.206.128@tcp101 [8/256/0/180] [13451.422082] LNet: Added LNI 192.168.206.128@tcp102 [8/256/0/180] [13452.194875] LNet: Added LNI 192.168.206.128@tcp103 [8/256/0/180] [13453.202526] LNet: Added LNI 192.168.206.128@tcp104 [8/256/0/180] [13454.048883] LNet: Added LNI 192.168.206.128@tcp105 [8/256/0/180] [13454.986618] LNet: Added LNI 192.168.206.128@tcp106 [8/256/0/180] [13455.910288] LNet: Added LNI 192.168.206.128@tcp107 [8/256/0/180] [13457.003270] LNet: Added LNI 192.168.206.128@tcp108 [8/256/0/180] [13458.171717] LNet: Added LNI 192.168.206.128@tcp109 [8/256/0/180] [13459.100746] LNet: Added LNI 192.168.206.128@tcp110 [8/256/0/180] [13459.949629] LNet: Added LNI 192.168.206.128@tcp111 [8/256/0/180] [13460.815527] LNet: Added LNI 192.168.206.128@tcp112 [8/256/0/180] [13461.683651] LNet: Added LNI 192.168.206.128@tcp113 [8/256/0/180] [13462.479824] LNet: Added LNI 192.168.206.128@tcp114 [8/256/0/180] [13463.265779] LNet: Added LNI 192.168.206.128@tcp115 [8/256/0/180] [13464.273097] LNet: Added LNI 192.168.206.128@tcp116 [8/256/0/180] [13465.177060] LNet: Added LNI 192.168.206.128@tcp117 [8/256/0/180] [13466.067468] LNet: Added LNI 192.168.206.128@tcp118 [8/256/0/180] [13466.859590] LNet: Added LNI 192.168.206.128@tcp119 [8/256/0/180] [13467.817496] LNet: Added LNI 192.168.206.128@tcp120 [8/256/0/180] [13468.657062] LNet: Added LNI 192.168.206.128@tcp121 [8/256/0/180] [13469.504731] LNet: Added LNI 192.168.206.128@tcp122 [8/256/0/180] [13470.394286] LNet: Added LNI 192.168.206.128@tcp123 [8/256/0/180] [13471.174700] LNet: Added LNI 192.168.206.128@tcp124 [8/256/0/180] [13472.168123] LNet: Added LNI 192.168.206.128@tcp125 [8/256/0/180] [13473.027736] LNet: Added LNI 192.168.206.128@tcp126 [8/256/0/180] [13473.947904] LNet: Added LNI 192.168.206.128@tcp127 [8/256/0/180] [13474.795416] LNet: Added LNI 192.168.206.128@tcp128 [8/256/0/180] [13475.718628] LNet: Added LNI 192.168.206.128@tcp129 [8/256/0/180] [13476.865043] LNet: Added LNI 192.168.206.128@tcp130 [8/256/0/180] [13477.784458] LNet: Added LNI 192.168.206.128@tcp131 [8/256/0/180] [13478.708661] LNet: Added LNI 192.168.206.128@tcp132 [8/256/0/180] [13479.705620] LNet: Added LNI 192.168.206.128@tcp133 [8/256/0/180] [13480.664396] LNet: Added LNI 192.168.206.128@tcp134 [8/256/0/180] [13481.645610] LNet: Added LNI 192.168.206.128@tcp135 [8/256/0/180] [13488.589256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13488.611969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13489.776189] Lustre: 463748:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.128@tcp: can't add NID 192.168.206.128@tcp130: rc = -75 [13489.860811] LNetError: 463771:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.128 [13489.905278] LNetError: Protocol error connecting to 192.168.206.128@tcp115 at host 192.168.206.128:988: is it running a compatible version of Lustre? [13489.926506] LNetError: Skipped 4 previous similar messages [13489.974757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13489.996040] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13490.069399] Lustre: lustre-MDT0000: new disk, initializing [13490.144303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13490.163468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13490.719549] LNet: 2 local NIs in recovery (showing 2): 192.168.206.128@tcp128, 192.168.206.128@tcp129 [13490.720447] LNetError: 463768:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.128 [13490.736472] LNetError: 463768:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [13490.747292] LNetError: Protocol error connecting to 192.168.206.128@tcp110 at host 192.168.206.128:988: is it running a compatible version of Lustre? [13494.834521] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [13505.281842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13505.425993] Lustre: 463763:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13505.478442] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13505.481642] Lustre: 464567:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.128@tcp: can't add NID 192.168.206.128@tcp130: rc = -75 [13505.486781] Lustre: Skipped 1 previous similar message [13505.493750] Lustre: 464567:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13505.528212] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13505.579609] Lustre: lustre-MDT0001: new disk, initializing [13505.693547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13505.717568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13505.726076] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13510.493327] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [13519.994252] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13520.416459] LNetError: 463776:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.128 [13520.438726] LNetError: Protocol error connecting to 192.168.206.128@tcp115 at host 192.168.206.128:988: is it running a compatible version of Lustre? [13523.489489] LNetError: 463780:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.128 [13523.495263] LNetError: 463780:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [13523.506403] LNetError: Protocol error connecting to 192.168.206.128@tcp126 at host 192.168.206.128:988: is it running a compatible version of Lustre? [13523.524309] LNetError: Skipped 1 previous similar message [13525.745746] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13528.615103] LNetError: 461207:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.128 [13528.621406] LNetError: Protocol error connecting to 192.168.206.128@tcp127 at host 192.168.206.128:988: is it running a compatible version of Lustre? [13532.716749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13533.058647] Lustre: lustre-OST0000: new disk, initializing [13533.069084] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13533.078130] Lustre: 465738:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13533.182905] Lustre: 465743:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.128@tcp: can't add NID 192.168.206.128@tcp130: rc = -75 [13533.192119] Lustre: 465743:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [13533.235273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13538.840292] Lustre: 466053:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.128@tcp: can't add NID 192.168.206.128@tcp130: rc = -75 [13538.857281] Lustre: 466053:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13538.934680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13538.948425] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13539.068284] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13539.217894] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [13540.899078] LNetError: 463771:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.128 [13540.910375] LNetError: 463771:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [13540.920050] LNetError: Protocol error connecting to 192.168.206.128@tcp127 at host 192.168.206.128:988: is it running a compatible version of Lustre? [13540.928075] LNetError: Skipped 3 previous similar messages [13547.644314] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13554.662732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13554.669297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13554.679494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13554.683382] Lustre: Skipped 1 previous similar message [13554.692660] Lustre: Skipped 1 previous similar message [13558.304793] LNetError: 463782:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.128 [13558.333217] LNetError: 463782:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [13558.354067] LNetError: Protocol error connecting to 192.168.206.128@tcp115 at host 192.168.206.128:988: is it running a compatible version of Lustre? [13558.377255] LNetError: Skipped 4 previous similar messages [13559.787478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13559.801775] Lustre: Skipped 1 previous similar message [13560.378364] Lustre: server umount lustre-OST0000 complete [13564.899902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13564.912319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13564.919046] Lustre: Skipped 1 previous similar message [13570.019612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13570.033647] Lustre: Skipped 1 previous similar message [13578.207787] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13578.460574] Lustre: server umount lustre-MDT0000 complete [13582.512298] LustreError: 463758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390388 with bad export cookie 5753573219200698811 [13582.521220] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13582.521498] LustreError: 463758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13583.046689] Lustre: server umount lustre-MDT0001 complete [13592.096707] LNetError: 463771:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.128 [13592.106836] LNetError: 463771:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [13592.114436] LNetError: Protocol error connecting to 192.168.206.128@tcp127 at host 192.168.206.128:988: is it running a compatible version of Lustre? [13592.132414] LNetError: Skipped 5 previous similar messages [13593.660916] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [13596.562806] Key type lgssc unregistered [13596.881211] LNet: 467654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13596.888368] LNetError: 467654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13596.895077] LNet: Removed LNI 192.168.206.128@tcp [13597.919288] LNet: Removed LNI 192.168.206.128@tcp100 [13597.925313] LNetError: 467654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13597.935911] LNetError: 467654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13597.948341] LNet: Removed LNI 192.168.206.128@tcp101 [13599.007693] LNet: Removed LNI 192.168.206.128@tcp102 [13599.018445] LNetError: 467654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13599.033709] LNetError: 467654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13599.042497] LNet: Removed LNI 192.168.206.128@tcp103 [13600.095458] LNet: Removed LNI 192.168.206.128@tcp104 [13601.119158] LNet: Removed LNI 192.168.206.128@tcp105 [13601.123714] LNetError: 467654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13601.129416] LNetError: 467654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13602.143209] LNet: Removed LNI 192.168.206.128@tcp106 [13603.167134] LNet: Removed LNI 192.168.206.128@tcp107 [13604.191174] LNet: Removed LNI 192.168.206.128@tcp108 [13605.215146] LNet: Removed LNI 192.168.206.128@tcp109 [13605.217326] LNetError: 467654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13605.228647] LNetError: 467654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [13605.238983] LNet: Removed LNI 192.168.206.128@tcp110 [13606.303360] LNet: Removed LNI 192.168.206.128@tcp111 [13607.327566] LNet: Removed LNI 192.168.206.128@tcp112 [13608.351580] LNet: Removed LNI 192.168.206.128@tcp113 [13609.377582] LNet: Removed LNI 192.168.206.128@tcp114 [13610.399194] LNet: Removed LNI 192.168.206.128@tcp115 [13611.423395] LNet: Removed LNI 192.168.206.128@tcp116 [13612.447168] LNet: Removed LNI 192.168.206.128@tcp117 [13613.471335] LNet: Removed LNI 192.168.206.128@tcp118 [13613.474964] LNetError: 467654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13613.494572] LNetError: 467654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13613.504082] LNet: Removed LNI 192.168.206.128@tcp119 [13614.559257] LNet: Removed LNI 192.168.206.128@tcp120 [13615.583208] LNet: Removed LNI 192.168.206.128@tcp121 [13616.607262] LNet: Removed LNI 192.168.206.128@tcp122 [13617.631154] LNet: Removed LNI 192.168.206.128@tcp123 [13618.655185] LNet: Removed LNI 192.168.206.128@tcp124 [13619.679138] LNet: Removed LNI 192.168.206.128@tcp125 [13620.703158] LNet: Removed LNI 192.168.206.128@tcp126 [13621.727159] LNet: Removed LNI 192.168.206.128@tcp127 [13622.751258] LNet: Removed LNI 192.168.206.128@tcp128 [13623.775296] LNet: Removed LNI 192.168.206.128@tcp129 [13623.780536] LNet: Removed LNI 192.168.206.128@tcp130 [13623.785519] LNet: Removed LNI 192.168.206.128@tcp131 [13623.789867] LNet: Removed LNI 192.168.206.128@tcp132 [13623.797876] LNet: Removed LNI 192.168.206.128@tcp133 [13623.801761] LNet: Removed LNI 192.168.206.128@tcp134 [13623.813918] LNet: Removed LNI 192.168.206.128@tcp135 [13624.728671] Key type .llcrypt unregistered [13624.730704] Key type ._llcrypt unregistered [13649.460584] Key type ._llcrypt registered [13649.463273] Key type .llcrypt registered [13649.597356] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [13667.678226] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [13668.620524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13668.640075] alg: No test for adler32 (adler32-zlib) [13669.926723] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [13670.217797] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [13671.919168] Key type lgssc registered [13673.371844] Lustre: Echo OBD driver; http://www.lustre.org/ [13709.696980] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 05:21:54 (1787390514) [13727.384932] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [13730.005822] Key type lgssc unregistered [13730.284734] LNet: 471740:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13730.293044] LNetError: 471740:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13730.326218] LNet: Removed LNI 192.168.206.128@tcp [13731.093519] Key type .llcrypt unregistered [13731.095370] Key type ._llcrypt unregistered [13747.626398] Key type ._llcrypt registered [13747.630584] Key type .llcrypt registered [13747.787576] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [13749.046310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13749.176202] alg: No test for adler32 (adler32-zlib) [13750.244878] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [13750.543603] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [13752.225402] Key type lgssc registered [13753.351553] Lustre: Echo OBD driver; http://www.lustre.org/ [13762.246309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13762.264234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13763.577553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13763.602656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13763.724690] Lustre: lustre-MDT0000: new disk, initializing [13763.792494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13763.806422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13768.196430] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [13778.417137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13778.530806] Lustre: 472956:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13778.575493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13778.579193] Lustre: Skipped 1 previous similar message [13778.640688] Lustre: lustre-MDT0001: new disk, initializing [13778.773468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13778.839578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13778.848666] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13782.981475] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [13791.582376] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13796.299704] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13802.273329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13802.495924] Lustre: lustre-OST0000: new disk, initializing [13802.503852] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13802.510815] Lustre: 474922:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13802.630342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13808.435416] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [13808.724066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13808.744549] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13808.850255] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13816.564844] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13820.027943] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing lnet_if_list [13825.534352] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13825.598649] LNet: Added LNI 192.168.206.128@tcp10 [8/256/0/180] [13830.915741] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13830.970881] LNet: Added LNI 192.168.206.128@tcp11 [8/256/0/180] [13835.933978] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13835.972962] LNet: Added LNI 192.168.206.128@tcp12 [8/256/0/180] [13840.562657] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13840.614470] LNet: Added LNI 192.168.206.128@tcp13 [8/256/0/180] [13845.811196] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13845.848504] LNet: Added LNI 192.168.206.128@tcp14 [8/256/0/180] [13852.684207] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13852.708452] LNet: Added LNI 192.168.206.128@tcp15 [8/256/0/180] [13857.797362] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13857.833551] LNet: Added LNI 192.168.206.128@tcp16 [8/256/0/180] [13863.788606] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13863.826282] LNet: Added LNI 192.168.206.128@tcp17 [8/256/0/180] [13869.090170] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13869.127145] LNet: Added LNI 192.168.206.128@tcp18 [8/256/0/180] [13873.935898] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13874.000559] LNet: Added LNI 192.168.206.128@tcp19 [8/256/0/180] [13879.360567] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13879.402298] LNet: Added LNI 192.168.206.128@tcp20 [8/256/0/180] [13884.439448] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13884.483129] LNet: Added LNI 192.168.206.128@tcp21 [8/256/0/180] [13889.917559] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13889.937986] LNet: Added LNI 192.168.206.128@tcp22 [8/256/0/180] [13896.385213] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13896.422347] LNet: Added LNI 192.168.206.128@tcp23 [8/256/0/180] [13901.753625] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13901.811710] LNet: Added LNI 192.168.206.128@tcp24 [8/256/0/180] [13907.522286] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13907.548960] LNet: Added LNI 192.168.206.128@tcp25 [8/256/0/180] [13912.842493] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13912.887375] LNet: Added LNI 192.168.206.128@tcp26 [8/256/0/180] [13919.147376] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13919.172620] LNet: Added LNI 192.168.206.128@tcp27 [8/256/0/180] [13924.658780] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13924.684637] LNet: Added LNI 192.168.206.128@tcp28 [8/256/0/180] [13929.714542] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13929.743826] LNet: Added LNI 192.168.206.128@tcp29 [8/256/0/180] [13935.230424] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13935.268932] LNet: Added LNI 192.168.206.128@tcp30 [8/256/0/180] [13941.143143] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13941.168933] LNet: Added LNI 192.168.206.128@tcp31 [8/256/0/180] [13947.281466] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13947.325917] LNet: Added LNI 192.168.206.128@tcp32 [8/256/0/180] [13953.150714] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13953.177904] LNet: Added LNI 192.168.206.128@tcp33 [8/256/0/180] [13959.542565] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13959.588251] LNet: Added LNI 192.168.206.128@tcp34 [8/256/0/180] [13966.304615] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13966.332250] LNet: Added LNI 192.168.206.128@tcp35 [8/256/0/180] [13972.835383] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13972.860770] LNet: Added LNI 192.168.206.128@tcp36 [8/256/0/180] [13979.979600] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13980.013434] LNet: Added LNI 192.168.206.128@tcp37 [8/256/0/180] [13986.346627] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13986.376685] LNet: Added LNI 192.168.206.128@tcp38 [8/256/0/180] [13993.587957] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13993.616164] LNet: Added LNI 192.168.206.128@tcp39 [8/256/0/180] [14000.405539] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14000.429645] LNet: Added LNI 192.168.206.128@tcp40 [8/256/0/180] [14006.689594] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14006.720375] LNet: Added LNI 192.168.206.128@tcp41 [8/256/0/180] [14012.826274] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14012.862686] LNet: Added LNI 192.168.206.128@tcp42 [8/256/0/180] [14018.815868] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14018.846798] LNet: Added LNI 192.168.206.128@tcp43 [8/256/0/180] [14025.175514] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14025.214344] LNet: Added LNI 192.168.206.128@tcp44 [8/256/0/180] [14032.164504] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14032.194158] LNet: Added LNI 192.168.206.128@tcp45 [8/256/0/180] [14038.519061] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14038.553648] LNet: Added LNI 192.168.206.128@tcp46 [8/256/0/180] [14046.269864] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14046.311444] LNet: Added LNI 192.168.206.128@tcp47 [8/256/0/180] [14054.010082] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14054.042075] LNet: Added LNI 192.168.206.128@tcp48 [8/256/0/180] [14061.545930] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14061.578921] LNet: Added LNI 192.168.206.128@tcp49 [8/256/0/180] [14068.524447] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14068.547766] LNet: Added LNI 192.168.206.128@tcp50 [8/256/0/180] [14086.112859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14086.118387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14086.127531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14088.677256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14088.679625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14088.691774] Lustre: Skipped 1 previous similar message [14088.704384] Lustre: Skipped 2 previous similar messages [14093.793786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14093.801228] Lustre: Skipped 2 previous similar messages [14098.399359] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14098.511628] Lustre: server umount lustre-MDT0000 complete [14101.602228] LustreError: 473750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390907 with bad export cookie 6772652729964938867 [14101.603986] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14101.615827] LustreError: 473750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14101.960487] Lustre: server umount lustre-MDT0001 complete [14119.173439] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [14119.391098] Lustre: 472226:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787390909/real 1787390909] req@0000000086a72d1a x1874214839811712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787390925 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14119.424621] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14121.440090] Lustre: 472228:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787390911/real 1787390911] req@0000000023a1e862 x1874214839811840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787390927 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14124.512303] Lustre: 472228:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787390914/real 1787390914] req@00000000e481d99d x1874214839811968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787390930 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14126.168592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14126.623478] LustreError: 472224:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000584bebf6 x1874214839812608/t0(0) o250->MGC192.168.206.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14126.967289] LustreError: 474916:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14126.999127] LustreError: 474916:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [14127.040887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14128.096829] Lustre: 472226:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787390917/real 1787390917] req@000000008a6f773a x1874214839812224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787390933 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14129.067519] LustreError: 474917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14129.068840] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14129.104128] LustreError: 474917:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14130.147576] LustreError: 499162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14130.174132] LustreError: 499162:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14131.020455] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [14134.176201] Lustre: 472226:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787390923/real 1787390923] req@000000000fcbcce3 x1874214839812352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787390939 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14135.278798] LustreError: 499161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14135.299992] LustreError: 499161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14140.386082] LustreError: 474917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14140.399898] LustreError: 474917:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14140.842812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14141.147133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14145.798521] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [14146.545838] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14152.723562] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14158.383521] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14177.248748] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14177.280489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14177.292758] Lustre: Skipped 1 previous similar message [14181.767739] Lustre: server umount lustre-OST0000 complete [14187.497197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14187.504818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14187.517548] Lustre: Skipped 1 previous similar message [14187.531822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14187.552187] Lustre: Skipped 1 previous similar message [14190.953247] Lustre: server umount lustre-MDT0000 complete [14192.609551] LustreError: 499166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14192.632731] LustreError: 499166:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14195.014478] LustreError: 472947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391000 with bad export cookie 6772652729964941443 [14195.015946] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14195.037823] LustreError: 472947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14195.523699] Lustre: server umount lustre-MDT0001 complete [14206.929511] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [14209.507722] Key type lgssc unregistered [14209.911416] LNet: 501939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14209.921858] LNetError: 501939:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14210.978228] LNet: Removed LNI 192.168.206.128@tcp [14210.984817] LNetError: 501939:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14211.000081] LNet: Removed LNI 192.168.206.128@tcp10 [14211.002796] LNet: Removed LNI 192.168.206.128@tcp11 [14211.008421] LNet: Removed LNI 192.168.206.128@tcp12 [14211.011247] LNet: Removed LNI 192.168.206.128@tcp13 [14211.013657] LNet: Removed LNI 192.168.206.128@tcp14 [14211.031836] LNet: Removed LNI 192.168.206.128@tcp15 [14211.037173] LNet: Removed LNI 192.168.206.128@tcp16 [14211.039526] LNet: Removed LNI 192.168.206.128@tcp17 [14211.044665] LNet: Removed LNI 192.168.206.128@tcp18 [14211.052858] LNet: Removed LNI 192.168.206.128@tcp19 [14211.056586] LNet: Removed LNI 192.168.206.128@tcp20 [14211.060560] LNet: Removed LNI 192.168.206.128@tcp21 [14211.067556] LNet: Removed LNI 192.168.206.128@tcp22 [14211.074074] LNet: Removed LNI 192.168.206.128@tcp23 [14211.080049] LNet: Removed LNI 192.168.206.128@tcp24 [14211.089461] LNet: Removed LNI 192.168.206.128@tcp25 [14211.094726] LNet: Removed LNI 192.168.206.128@tcp26 [14211.099760] LNet: Removed LNI 192.168.206.128@tcp27 [14211.102409] LNet: Removed LNI 192.168.206.128@tcp28 [14211.107499] LNet: Removed LNI 192.168.206.128@tcp29 [14211.113146] LNet: Removed LNI 192.168.206.128@tcp30 [14211.117785] LNet: Removed LNI 192.168.206.128@tcp31 [14211.125151] LNet: Removed LNI 192.168.206.128@tcp32 [14211.128138] LNet: Removed LNI 192.168.206.128@tcp33 [14211.132025] LNet: Removed LNI 192.168.206.128@tcp34 [14211.136946] LNet: Removed LNI 192.168.206.128@tcp35 [14211.143516] LNet: Removed LNI 192.168.206.128@tcp36 [14211.148559] LNet: Removed LNI 192.168.206.128@tcp37 [14211.157186] LNet: Removed LNI 192.168.206.128@tcp38 [14211.164117] LNet: Removed LNI 192.168.206.128@tcp39 [14211.168566] LNet: Removed LNI 192.168.206.128@tcp40 [14211.174579] LNet: Removed LNI 192.168.206.128@tcp41 [14211.180080] LNet: Removed LNI 192.168.206.128@tcp42 [14211.184740] LNet: Removed LNI 192.168.206.128@tcp43 [14211.192734] LNet: Removed LNI 192.168.206.128@tcp44 [14211.198182] LNet: Removed LNI 192.168.206.128@tcp45 [14211.205701] LNet: Removed LNI 192.168.206.128@tcp46 [14211.213392] LNet: Removed LNI 192.168.206.128@tcp47 [14211.219075] LNet: Removed LNI 192.168.206.128@tcp48 [14211.224528] LNet: Removed LNI 192.168.206.128@tcp49 [14211.234198] LNet: Removed LNI 192.168.206.128@tcp50 [14212.024230] Key type .llcrypt unregistered [14212.027650] Key type ._llcrypt unregistered [14231.957523] Key type ._llcrypt registered [14231.960490] Key type .llcrypt registered [14232.089341] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [14248.540938] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [14249.471562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14249.559622] alg: No test for adler32 (adler32-zlib) [14250.573598] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [14250.873908] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [14252.583177] Key type lgssc registered [14254.088207] Lustre: Echo OBD driver; http://www.lustre.org/ [14288.975166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14289.024531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14290.377338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14290.414538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14290.503935] Lustre: lustre-MDT0000: new disk, initializing [14290.624866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14290.678241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14295.945468] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [14306.590374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14306.742549] Lustre: 505140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14306.818143] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14306.821282] Lustre: Skipped 1 previous similar message [14306.879685] Lustre: lustre-MDT0001: new disk, initializing [14306.972230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14307.039039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14307.060776] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14311.166813] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [14321.328839] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14326.627232] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14333.863898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14334.138780] Lustre: *** cfs_fail_loc=90f, val=0*** [14334.257636] Lustre: lustre-OST0000: new disk, initializing [14334.261765] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14334.269867] Lustre: 507154:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14334.378535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14340.552436] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [14344.269969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14344.286216] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14344.386788] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14348.588792] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14360.033757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14360.038227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14360.044664] LustreError: Skipped 1 previous similar message [14360.059046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14360.062593] Lustre: Skipped 1 previous similar message [14365.153357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14365.174902] Lustre: Skipped 2 previous similar messages [14369.247493] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14369.409772] Lustre: server umount lustre-OST0000 complete [14373.860477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14373.872444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14373.906684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14375.393859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14380.512631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14380.518874] Lustre: Skipped 3 previous similar messages [14385.634528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14385.640326] Lustre: Skipped 1 previous similar message [14387.170855] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14387.319982] Lustre: server umount lustre-MDT0000 complete [14390.408024] LustreError: 505132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391196 with bad export cookie 340946495734926575 [14390.414593] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14390.420396] LustreError: 505132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14390.737179] Lustre: server umount lustre-MDT0001 complete [14399.602629] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [14402.159472] Key type lgssc unregistered [14402.510944] LNet: 509061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14402.522701] LNetError: 509061:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14402.547158] LNet: Removed LNI 192.168.206.128@tcp [14403.321593] Key type .llcrypt unregistered [14403.324595] Key type ._llcrypt unregistered [14421.011900] Key type ._llcrypt registered [14421.019692] Key type .llcrypt registered [14421.134162] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [14422.150521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14422.281767] alg: No test for adler32 (adler32-zlib) [14423.313674] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [14423.540969] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [14425.199810] Key type lgssc registered [14426.230443] Lustre: Echo OBD driver; http://www.lustre.org/ [14434.138084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14434.177033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14435.815618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14440.690757] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [14446.051182] LustreError: 510327:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14446.087461] LustreError: 510327:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [14449.022899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14449.335145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14454.067691] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [14461.626449] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14467.834480] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14474.795211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14475.161069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14481.994263] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [14488.872675] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14499.425144] loop0: detected capacity change from 0 to 204800000 [14499.658715] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14499.675767] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14499.686740] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14505.099265] loop0: detected capacity change from 0 to 204800000 [14505.327123] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14505.343643] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14505.359303] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14510.742686] loop0: detected capacity change from 0 to 204800000 [14516.685594] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14516.850328] Lustre: fs15246-OST0000: new disk, initializing [14516.853724] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14516.858793] Lustre: 513922:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14516.921543] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14523.871144] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [14533.755933] loop1: detected capacity change from 0 to 204800000 [14539.680406] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14539.818866] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14539.874826] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14539.963640] Lustre: fs15246-MDT0000: new disk, initializing [14540.182324] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14540.209248] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14540.345193] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14541.408464] Lustre: 515045:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000075c61c78 x1874215545385728/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787391363 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14545.344098] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [14550.498726] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14550.510397] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14550.533739] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14568.418883] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14568.459223] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14573.541425] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14573.770151] Lustre: server umount fs15246-MDT0000 complete [14578.670642] LustreError: 510322:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14582.166623] Lustre: server umount fs15246-OST0000 complete [14598.113746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14598.129336] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14598.146880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14599.138600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14599.146966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14603.020031] Lustre: server umount lustre-OST0000 complete [14608.352964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14608.357878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14608.381028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14608.390530] Lustre: Skipped 2 previous similar messages [14614.496160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14614.507513] Lustre: Skipped 3 previous similar messages [14621.152973] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14621.386211] Lustre: server umount lustre-MDT0000 complete [14624.736184] LustreError: 510326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14624.773641] LustreError: 510326:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14625.212844] LustreError: 515039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391431 with bad export cookie 16776980823654458461 [14625.217696] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14625.222554] LustreError: 515039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14625.588846] Lustre: server umount lustre-MDT0001 complete [14637.268748] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [14640.050375] Key type lgssc unregistered [14640.279942] LNet: 518142:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14640.288932] LNetError: 518142:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14640.316404] LNet: Removed LNI 192.168.206.128@tcp [14641.034530] Key type .llcrypt unregistered [14641.041429] Key type ._llcrypt unregistered [14660.621718] Key type ._llcrypt registered [14660.623828] Key type .llcrypt registered [14660.711315] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [14661.628109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14661.727667] alg: No test for adler32 (adler32-zlib) [14662.835262] Lustre: Lustre: Build Version: 2.17.54_117_g8b0669c [14663.128890] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [14664.855143] Key type lgssc registered [14666.115321] Lustre: Echo OBD driver; http://www.lustre.org/ [14707.282322] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [14717.591150] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [14755.420814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14755.473208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14756.796479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14756.841884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14756.968398] Lustre: lustre-MDT0000: new disk, initializing [14757.100856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14757.133275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14761.652435] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [14772.750589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14772.935535] Lustre: 522612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14773.009450] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14773.013325] Lustre: Skipped 1 previous similar message [14773.078604] Lustre: lustre-MDT0001: new disk, initializing [14773.170603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14773.205876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14773.223192] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14777.868587] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [14787.111449] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14792.561435] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14800.167991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14800.417898] Lustre: lustre-OST0000: new disk, initializing [14800.421177] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14800.427528] Lustre: 524582:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14800.529849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14806.550818] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [14809.130233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14809.141279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14809.242312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14816.073531] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14820.909483] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14821.081682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14824.742979] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14824.893933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14830.051653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14830.055683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14830.075989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14830.086903] Lustre: Skipped 1 previous similar message [14832.630878] Lustre: server umount lustre-OST0000 complete [14839.779374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14839.784631] LustreError: Skipped 1 previous similar message [14839.787691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14839.795360] Lustre: Skipped 1 previous similar message [14839.801451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14844.277899] Lustre: server umount lustre-MDT0000 complete [14849.637611] LustreError: 523406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391655 with bad export cookie 5606917296548876946 [14849.647434] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14849.649996] LustreError: 523406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14850.202371] Lustre: server umount lustre-MDT0001 complete [14875.732777] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [14887.267899] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [14920.544861] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14659 sec ========= 05:42:04 (1787391724) [14922.281863] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:42:06 (1787391726) === [14925.988903] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:42:10 (1787391730) === [14966.485979] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [14969.083625] Key type lgssc unregistered [14969.268532] LNet: 531225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14969.272942] LNetError: 531225:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14969.281555] LNet: Removed LNI 192.168.206.128@tcp [14969.934247] Key type .llcrypt unregistered [14969.936393] Key type ._llcrypt unregistered