[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 588745063 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524576K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002353] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.006737] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009125] LSM: Security Framework initializing [ 0.010055] Yama: becoming mindful. [ 0.011042] SELinux: Initializing. [ 0.012077] *** VALIDATE selinux *** [ 0.020109] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024508] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025119] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027089] *** VALIDATE tmpfs *** [ 0.029127] *** VALIDATE proc *** [ 0.030206] *** VALIDATE cgroup *** [ 0.031010] *** VALIDATE cgroup2 *** [ 0.033036] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036032] Spectre V2 : User space: Vulnerable [ 0.037007] Speculative Store Bypass: Vulnerable [ 0.040192] debug: unmapping init [mem 0xffffffffbb859000-0xffffffffbb860fff] [ 0.042168] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043510] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044018] ... version: 2 [ 0.045012] ... bit width: 48 [ 0.045895] ... generic registers: 4 [ 0.046013] ... value mask: 0000ffffffffffff [ 0.047008] ... max period: 00007fffffffffff [ 0.048007] ... fixed-purpose events: 3 [ 0.048906] ... event mask: 000000070000000f [ 0.049258] rcu: Hierarchical SRCU implementation. [ 0.051137] smp: Bringing up secondary CPUs ... [ 0.052481] x86: Booting SMP configuration: [ 0.053023] .... node #0, CPUs: #1 #2 #3 [ 0.055757] smp: Brought up 1 node, 4 CPUs [ 0.056983] smpboot: Max logical packages: 1 [ 0.057008] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.129024] node 0 deferred pages initialised in 70ms [ 0.133152] devtmpfs: initialized [ 0.134231] x86/mm: Memory block size: 128MB [ 0.137413] gcov: version magic: 0x41383552 [ 0.141280] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.146096] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.148510] pinctrl core: initialized pinctrl subsystem [ 0.151331] [ 0.151924] ************************************************************* [ 0.153019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156021] ** ** [ 0.159020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161018] ** ** [ 0.163018] ** This means that this kernel is built to expose internal ** [ 0.165027] ** IOMMU data structures, which may compromise security on ** [ 0.167019] ** your system. ** [ 0.169018] ** ** [ 0.171020] ** If you see this message and you are not debugging the ** [ 0.174016] ** kernel, report this immediately to your vendor! ** [ 0.176016] ** ** [ 0.178018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.180017] ************************************************************* [ 0.182773] NET: Registered protocol family 16 [ 0.185464] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.188056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.190066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.194020] cpuidle: using governor menu [ 0.195679] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.197472] PCI: Using configuration type 1 for base access [ 0.200144] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.210147] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.211020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.213041] cryptd: max_cpu_qlen set to 1000 [ 0.215233] ACPI: Added _OSI(Module Device) [ 0.217019] ACPI: Added _OSI(Processor Device) [ 0.220022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.221019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.226148] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.233586] ACPI: Interpreter enabled [ 0.235060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.237011] ACPI: Using IOAPIC for interrupt routing [ 0.238132] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.242402] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.253964] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.257044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.260018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.263084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.269000] acpiphp: Slot [2] registered [ 0.270165] acpiphp: Slot [5] registered [ 0.272113] acpiphp: Slot [6] registered [ 0.274140] acpiphp: Slot [7] registered [ 0.275158] acpiphp: Slot [8] registered [ 0.277200] acpiphp: Slot [9] registered [ 0.278142] acpiphp: Slot [10] registered [ 0.280080] acpiphp: Slot [3] registered [ 0.281085] acpiphp: Slot [4] registered [ 0.283129] acpiphp: Slot [11] registered [ 0.284161] acpiphp: Slot [12] registered [ 0.286094] acpiphp: Slot [13] registered [ 0.288108] acpiphp: Slot [14] registered [ 0.289088] acpiphp: Slot [15] registered [ 0.290112] acpiphp: Slot [16] registered [ 0.292115] acpiphp: Slot [17] registered [ 0.294078] acpiphp: Slot [18] registered [ 0.296121] acpiphp: Slot [19] registered [ 0.297099] acpiphp: Slot [20] registered [ 0.299100] acpiphp: Slot [21] registered [ 0.300093] acpiphp: Slot [22] registered [ 0.302116] acpiphp: Slot [23] registered [ 0.303088] acpiphp: Slot [24] registered [ 0.305086] acpiphp: Slot [25] registered [ 0.306082] acpiphp: Slot [26] registered [ 0.308086] acpiphp: Slot [27] registered [ 0.309092] acpiphp: Slot [28] registered [ 0.311110] acpiphp: Slot [29] registered [ 0.312093] acpiphp: Slot [30] registered [ 0.314075] acpiphp: Slot [31] registered [ 0.315066] PCI host bridge to bus 0000:00 [ 0.317019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.319024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.322024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.325024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.327019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.330026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.332157] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.335053] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.339281] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.349782] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.353015] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.356020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.359024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.361018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.365328] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.367854] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.371053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.375274] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.380017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.391932] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.397074] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.404869] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.416017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.425016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.446015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.459792] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.471016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.478016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.494017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.505931] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.517014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.522013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.545019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.573000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.601022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.631018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.649017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.661990] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.672017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.679016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.707018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.716951] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.729016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.739017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.758019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.775849] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.778424] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.782475] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.785397] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.788225] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.793649] iommu: Default domain type: Passthrough [ 0.795455] SCSI subsystem initialized [ 0.797152] ACPI: bus type USB registered [ 0.799110] usbcore: registered new interface driver usbfs [ 0.801122] usbcore: registered new interface driver hub [ 0.803070] usbcore: registered new device driver usb [ 0.804407] pps_core: LinuxPPS API ver. 1 registered [ 0.807015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.810061] PTP clock support registered [ 0.812074] EDAC MC: Ver: 3.0.0 [ 0.813490] PCI: Using ACPI for IRQ routing [ 0.815810] NetLabel: Initializing [ 0.817013] NetLabel: domain hash size = 128 [ 0.818008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.820094] NetLabel: unlabeled traffic allowed by default [ 0.822040] vgaarb: loaded [ 0.823261] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.825015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.832004] clocksource: Switched to clocksource kvm-clock [ 0.953296] VFS: Disk quotas dquot_6.6.0 [ 0.954557] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.957355] *** VALIDATE ramfs *** [ 0.958817] *** VALIDATE hugetlbfs *** [ 0.960406] pnp: PnP ACPI init [ 0.962990] pnp: PnP ACPI: found 6 devices [ 0.980834] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.984919] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.987240] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.989633] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.992284] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.994752] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.998873] NET: Registered protocol family 2 [ 1.001130] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.006434] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.010037] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.015335] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.021388] TCP: Hash tables configured (established 65536 bind 65536) [ 1.025762] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.030113] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.033285] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.035548] NET: Registered protocol family 1 [ 1.037680] RPC: Registered named UNIX socket transport module. [ 1.039731] RPC: Registered udp transport module. [ 1.040893] RPC: Registered tcp transport module. [ 1.042061] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.043878] NET: Registered protocol family 44 [ 1.044953] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.046893] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.049053] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.051577] PCI: CLS 0 bytes, default 64 [ 1.053722] Unpacking initramfs... [ 2.400048] debug: unmapping init [mem 0xffff9b9c7cc54000-0xffff9b9c7ffbffff] [ 2.404114] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.406541] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.409493] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.925062] Initialise system trusted keyrings [ 2.926713] Key type blacklist registered [ 2.928614] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.941492] zbud: loaded [ 2.945644] *** VALIDATE nfs *** [ 2.947162] *** VALIDATE nfs4 *** [ 2.949030] pstore: using deflate compression [ 2.953139] Platform Keyring initialized [ 3.050441] NET: Registered protocol family 38 [ 3.051987] Key type asymmetric registered [ 3.052845] Asymmetric key parser 'x509' registered [ 3.054214] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.056252] io scheduler mq-deadline registered [ 3.057350] io scheduler kyber registered [ 3.058541] io scheduler bfq registered [ 3.060504] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.063641] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.066682] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.069673] ACPI: Power Button [PWRF] [ 3.074891] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.080883] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.118499] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.135665] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.161063] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.192098] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.221441] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.226471] Non-volatile memory driver v1.3 [ 3.228213] Linux agpgart interface v0.103 [ 3.257032] virtio_blk virtio1: [vda] 150672 512-byte logical blocks (77.1 MB/73.6 MiB) [ 3.259901] vda: detected capacity change from 0 to 77144064 [ 3.274497] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.278409] vdb: detected capacity change from 0 to 1073741824 [ 3.299244] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.302577] vdc: detected capacity change from 0 to 2621440000 [ 3.320774] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.324221] vdd: detected capacity change from 0 to 2621440000 [ 3.340281] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.342823] vde: detected capacity change from 0 to 4294967296 [ 3.361851] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.364899] vdf: detected capacity change from 0 to 4294967296 [ 3.374220] libphy: Fixed MDIO Bus: probed [ 3.391555] usbcore: registered new interface driver usbserial_generic [ 3.394408] usbserial: USB Serial support registered for generic [ 3.397604] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.403458] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.405540] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.408596] mousedev: PS/2 mouse device common for all mice [ 3.412468] rtc_cmos 00:05: RTC can wake from S4 [ 3.415987] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.416217] rtc_cmos 00:05: registered as rtc0 [ 3.421246] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.421475] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.426738] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.427682] intel_pstate: CPU model not supported [ 3.432707] hid: raw HID events driver (C) Jiri Kosina [ 3.435033] usbcore: registered new interface driver usbhid [ 3.437187] usbhid: USB HID core driver [ 3.438947] drop_monitor: Initializing network drop monitor service [ 3.441540] Initializing XFRM netlink socket [ 3.443897] NET: Registered protocol family 10 [ 3.446992] Segment Routing with IPv6 [ 3.448528] NET: Registered protocol family 17 [ 3.450568] mpls_gso: MPLS GSO support [ 3.455417] RAS: Correctable Errors collector initialized. [ 3.457469] AVX version of gcm_enc/dec engaged. [ 3.458994] AES CTR mode by8 optimization enabled [ 3.540802] sched_clock: Marking stable (3540759152, 0)->(4432612832, -891853680) [ 3.545063] registered taskstats version 1 [ 3.547114] Loading compiled-in X.509 certificates [ 3.549214] zswap: loaded using pool lzo/zbud [ 3.575415] Key type big_key registered [ 3.588368] Key type encrypted registered [ 3.590317] ima: No TPM chip found, activating TPM-bypass! [ 3.592450] ima: Allocated hash algorithm: sha1 [ 3.594223] ima: No architecture policies found [ 3.596278] evm: Initialising EVM extended attributes: [ 3.598065] evm: security.selinux [ 3.599069] evm: security.ima [ 3.600034] evm: security.capability [ 3.601109] evm: HMAC attrs: 0x1 [ 3.603533] rtc_cmos 00:05: setting system clock to 2026-09-30 03:35:25 UTC (1790739325) [ 3.609636] debug: unmapping init [mem 0xffffffffbc803000-0xffffffffbc9fffff] [ 3.612298] debug: unmapping init [mem 0xffffffffbb582000-0xffffffffbb858fff] [ 3.624119] Write protecting the kernel read-only data: 28672k [ 3.627246] debug: unmapping init [mem 0xffffffffb9c03000-0xffffffffb9dfffff] [ 3.629953] debug: unmapping init [mem 0xffffffffba514000-0xffffffffba5fffff] [ 3.669289] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.677738] systemd[1]: Detected virtualization kvm. [ 3.679452] systemd[1]: Detected architecture x86-64. [ 3.681441] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.707240] systemd[1]: No hostname configured. [ 3.708682] systemd[1]: Set hostname to . [ 3.710573] random: systemd: uninitialized urandom read (16 bytes read) [ 3.713176] systemd[1]: Initializing machine ID from random generator. [ 3.754240] random: ln: uninitialized urandom read (6 bytes read) [ 3.855363] random: systemd: uninitialized urandom read (16 bytes read) [ 3.858138] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.862902] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.866509] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Timers. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.539933] device-mapper: uevent: version 1.0.3 [ 4.542343] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.244246] random: fast init done [ 5.274238] virtio_net virtio0 ens2: renamed from eth0 [ 5.367417] scsi host0: ata_piix [ 5.391506] scsi host1: ata_piix [ 5.393213] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.395920] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.995043] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.105793] random: crng init done [ 10.107323] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.560210] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.642180] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.849843] SELinux: Disabled at runtime. [ 11.910196] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.919060] systemd[1]: Detected virtualization kvm. [ 11.920236] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.364068] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.367427] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.370644] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.374475] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.376906] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.385434] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.395590] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [FAILED] [ 12.525124] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.758861] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.079301] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.096055] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.211406] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.228930] EDAC sbridge: Ver: 1.1.2 [ 14.745284] Key type dns_resolver registered [ 15.047391] NFS: Registering the id_resolver key type [ 15.049403] Key type id_resolver registered [ 15.050923] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg251-server login: [ 50.479636] libcfs: loading out-of-tree module taints kernel. [ 50.546366] Key type ._llcrypt registered [ 50.549390] Key type .llcrypt registered [ 50.670948] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 68.227251] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 69.724731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 69.735209] alg: No test for adler32 (adler32-zlib) [ 71.420792] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 72.378117] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 74.257105] Key type lgssc registered [ 76.207138] Lustre: Echo OBD driver; http://www.lustre.org/ [ 96.321970] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 140.575488] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 154.723895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 154.769170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 155.940011] hrtimer: interrupt took 8996139 ns [ 156.184264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 156.228354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 156.331241] Lustre: lustre-MDT0000: new disk, initializing [ 156.456540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 156.474647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 161.743458] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.451923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 178.602785] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 178.639563] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 178.644867] Lustre: Skipped 1 previous similar message [ 178.821379] Lustre: lustre-MDT0001: new disk, initializing [ 178.885588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 178.928931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 178.942268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 183.734787] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.919684] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 199.272760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 199.647947] Lustre: lustre-OST0000: new disk, initializing [ 199.657930] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 199.664935] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 199.673806] Lustre: 8454:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 199.752689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.856374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 204.871813] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 204.914395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 207.104399] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.245255] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 223.402871] Lustre: lustre-OST0001: new disk, initializing [ 223.406465] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 223.412800] Lustre: 9526:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 223.485329] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 230.077113] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.565431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 233.581130] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 233.659057] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 242.287688] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 248.938038] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 255.999202] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing check_logdir /tmp/testlogs/ [ 259.741403] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing yml_node [ 266.062578] Lustre: DEBUG MARKER: Client: 2.17.59.3 [ 268.632800] Lustre: DEBUG MARKER: MDS: 2.17.59.3 [ 271.228595] Lustre: DEBUG MARKER: OSS: 2.17.59.3 [ 273.316090] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 29 23:39:52 EDT 2026 [ 292.176432] Lustre: DEBUG MARKER: only running tests: 33-75 [ 293.984791] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 296.146383] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 312.291956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 312.304782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.330937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 315.877588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 315.879856] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.891820] Lustre: Skipped 1 previous similar message [ 315.913548] Lustre: Skipped 1 previous similar message [ 318.031364] Lustre: server umount lustre-MDT0000 complete [ 322.192535] LustreError: 8467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790739644 with bad export cookie 1294837216527654007 [ 322.193615] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 322.200996] LustreError: 8467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 326.111868] LustreError: 6522:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 326.112149] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.118248] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 326.118258] Lustre: Skipped 2 previous similar messages [ 326.125591] LustreError: 6522:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 326.196574] Lustre: Skipped 2 previous similar messages [ 327.136646] LustreError: 6521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 327.151369] LustreError: 6521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 328.987346] Lustre: server umount lustre-MDT0001 complete [ 333.906437] Lustre: server umount lustre-OST0000 complete [ 338.134522] Lustre: server umount lustre-OST0001 complete [ 345.861387] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 355.017600] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 391.028926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 391.420719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 391.507713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 391.636216] Lustre: lustre-MDT0000: new disk, initializing [ 391.751623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 391.781228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 396.986886] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 407.942086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 408.036799] Lustre: 15126:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 408.085854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 408.089117] Lustre: Skipped 1 previous similar message [ 408.170766] Lustre: lustre-MDT0001: new disk, initializing [ 408.270913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 408.314178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 408.332537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 413.582709] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 423.543885] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 428.977491] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 436.515630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 437.025477] Lustre: lustre-OST0000: new disk, initializing [ 437.032441] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 437.043986] Lustre: 17094:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 437.172517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 439.054858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 439.080945] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 439.228579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 444.582803] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 456.293089] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 461.492846] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 461.774483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 466.042465] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 466.318949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 469.999510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 470.002614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 470.016170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 470.040602] Lustre: Skipped 4 previous similar messages [ 474.502793] Lustre: server umount lustre-OST0000 complete [ 480.226212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 480.234650] Lustre: Skipped 1 previous similar message [ 480.235672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 480.236945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 480.245871] Lustre: Skipped 1 previous similar message [ 484.681897] Lustre: server umount lustre-MDT0000 complete [ 485.345113] LustreError: 15136:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.359719] LustreError: 15136:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 488.335884] LustreError: 15117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790739810 with bad export cookie 1294837216527655547 [ 488.346475] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 488.351629] LustreError: 15117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 488.744928] Lustre: server umount lustre-MDT0001 complete [ 493.000937] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 23:43:32 (1790739812) [ 495.842944] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 497.467623] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 23:43:37 (1790739817) [ 504.122895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 504.658555] LustreError: 19503:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.751204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 509.175390] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 509.922287] LustreError: 19504:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 519.067436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 524.490186] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 532.526280] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 538.833978] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 547.816523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 555.157930] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 563.597124] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 574.946036] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 574.955571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 574.962336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 574.978179] Lustre: Skipped 1 previous similar message [ 575.005722] Lustre: Skipped 1 previous similar message [ 580.816555] Lustre: server umount lustre-OST0000 complete [ 591.382591] Lustre: server umount lustre-MDT0000 complete [ 595.425490] LustreError: 19500:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.447861] LustreError: 19500:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 595.547469] LustreError: 19484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790739917 with bad export cookie 1294837216527656429 [ 595.557887] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 595.567774] LustreError: 19484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 595.951367] Lustre: server umount lustre-MDT0001 complete [ 609.061805] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 611.824133] Key type lgssc unregistered [ 612.198118] LNet: 23035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 612.208921] LNetError: 23035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 612.233740] LNet: Removed LNI 192.168.202.151@tcp [ 613.470214] Key type .llcrypt unregistered [ 613.472875] Key type ._llcrypt unregistered [ 638.282584] Key type ._llcrypt registered [ 638.284608] Key type .llcrypt registered [ 638.405289] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 639.856553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 639.883029] alg: No test for adler32 (adler32-zlib) [ 641.066504] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 641.482367] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 643.143283] Key type lgssc registered [ 644.465768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 653.005544] loop: module loaded [ 653.109408] loop0: detected capacity change from 0 to 204800000 [ 653.430203] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 653.443889] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 653.828461] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 660.229183] loop0: detected capacity change from 0 to 204800000 [ 660.485790] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 660.499872] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 660.510678] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 665.646590] loop0: detected capacity change from 0 to 204800000 [ 671.234460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 671.254137] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.548042] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 672.634280] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 672.753996] Lustre: lustre-MDT0000: new disk, initializing [ 672.917335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 672.962900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 677.576522] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 687.040256] loop1: detected capacity change from 0 to 204800000 [ 694.806961] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 695.249474] Lustre: lustre-OST07c6: new disk, initializing [ 695.256088] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 695.262169] Lustre: Skipped 1 previous similar message [ 695.270980] Lustre: 26239:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 695.398288] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 702.451893] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 704.118429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 704.128353] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 704.175894] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 707.206772] Lustre: 25225:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 707.214422] Lustre: 25225:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 707.218976] Lustre: 25225:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 707.228145] Lustre: 25225:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 707.239304] Lustre: 25225:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 707.246824] Lustre: 25225:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 724.353845] Lustre: 25226:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 724.367787] Lustre: 25226:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 724.380255] Lustre: 25226:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 724.400797] Lustre: 25226:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 724.417739] Lustre: 25226:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 724.425704] Lustre: 25226:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 729.883790] Lustre: 25226:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 729.898244] Lustre: 25226:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 729.905153] Lustre: 25226:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 729.916650] Lustre: 25226:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 729.922556] Lustre: 25226:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 729.930674] Lustre: 25226:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 729.941691] Lustre: 25226:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 729.953464] Lustre: 25226:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 729.964252] Lustre: 25226:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 729.977373] Lustre: 25226:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 729.992256] Lustre: 25226:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 730.004506] Lustre: 25226:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 733.376684] Lustre: 25227:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 733.398179] Lustre: 25227:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 733.404566] Lustre: 25227:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 733.413852] Lustre: 25227:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 733.425486] Lustre: 25227:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 733.432685] Lustre: 25227:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 733.440742] Lustre: 25227:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 733.449793] Lustre: 25227:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 733.457510] Lustre: 25227:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 733.468853] Lustre: 25227:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 733.480916] Lustre: 25227:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 733.486252] Lustre: 25227:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 738.096575] Lustre: 25227:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 738.101630] Lustre: 25227:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 738.106309] Lustre: 25227:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 738.110850] Lustre: 25227:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 738.115200] Lustre: 25227:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 738.119652] Lustre: 25227:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 738.124090] Lustre: 25227:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 738.128695] Lustre: 25227:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 738.135384] Lustre: 25227:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 738.140248] Lustre: 25227:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 738.144110] Lustre: 25227:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 738.149499] Lustre: 25227:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 750.431665] Lustre: 25225:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 750.439385] Lustre: 25225:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 5 previous similar messages [ 750.445790] Lustre: 25225:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 750.452186] Lustre: 25225:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 750.458402] Lustre: 25225:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 750.462731] Lustre: 25225:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 750.468607] Lustre: 25225:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 750.474074] Lustre: 25225:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 750.479206] Lustre: 25225:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 750.492578] Lustre: 25225:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 750.505083] Lustre: 25225:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 750.515568] Lustre: 25225:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 806.369785] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 806.390474] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 808.599579] Lustre: server umount lustre-OST07c6 complete [ 819.286211] Lustre: server umount lustre-MDT0000 complete [ 835.713400] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 838.292417] Key type lgssc unregistered [ 838.633608] LNet: 28793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.640143] LNetError: 28793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.658242] LNet: Removed LNI 192.168.202.151@tcp [ 839.643887] Key type .llcrypt unregistered [ 839.646754] Key type ._llcrypt unregistered [ 857.461968] Key type ._llcrypt registered [ 857.470755] Key type .llcrypt registered [ 857.572688] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 859.030156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 859.152608] alg: No test for adler32 (adler32-zlib) [ 860.225328] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 860.493978] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 862.208457] Key type lgssc registered [ 863.421676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 871.436461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 871.456180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 872.881569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 877.549880] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 882.146891] LustreError: 30080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 882.176108] LustreError: 30080:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 885.776804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 886.298163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 890.594577] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 897.665502] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 904.089860] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 910.734777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 911.241057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 917.368348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 917.580646] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 925.447294] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 943.078657] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 943.082531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 943.093936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 943.118996] Lustre: Skipped 1 previous similar message [ 945.774319] Lustre: server umount lustre-OST0000 complete [ 952.814362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 952.819544] LustreError: Skipped 1 previous similar message [ 952.833036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 952.856176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 952.864413] Lustre: Skipped 1 previous similar message [ 956.084494] Lustre: server umount lustre-MDT0000 complete [ 958.438207] LustreError: 30080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 958.471330] LustreError: 30080:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 960.939040] LustreError: 30060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790740282 with bad export cookie 16793021649476784749 [ 960.944445] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 960.948498] LustreError: 30060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 961.368916] Lustre: server umount lustre-MDT0001 complete [ 972.308891] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 975.975053] Key type lgssc unregistered [ 976.415563] LNet: 33815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 976.423644] LNetError: 33815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 976.444391] LNet: Removed LNI 192.168.202.151@tcp [ 977.746577] Key type .llcrypt unregistered [ 977.748749] Key type ._llcrypt unregistered [ 1003.793262] Key type ._llcrypt registered [ 1003.795308] Key type .llcrypt registered [ 1003.897257] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1004.470260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1004.499256] alg: No test for adler32 (adler32-zlib) [ 1005.546051] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 1005.749984] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 1007.391227] Key type lgssc registered [ 1008.666756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1016.980931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1017.027236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1018.535991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1024.205371] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1029.099743] LustreError: 35100:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1029.128875] LustreError: 35100:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1034.227667] LustreError: 35101:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1035.197505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1035.663489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1041.575282] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1050.814778] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1057.807721] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1064.933142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1065.184227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1072.163343] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1079.879601] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1088.767608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1104.346565] Lustre: server umount lustre-OST0000 complete [ 1109.474413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1109.477642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1109.501145] Lustre: Skipped 1 previous similar message [ 1109.513208] Lustre: Skipped 1 previous similar message [ 1114.606401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1114.619621] Lustre: Skipped 1 previous similar message [ 1118.763156] Lustre: server umount lustre-MDT0000 complete [ 1119.726501] LustreError: 35101:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1119.755766] LustreError: 35101:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1120.736547] Lustre: 34402:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790740426/real 1790740426] req@00000000f9d906c0 x1877727329156736/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790740442 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1120.767385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.850762] Lustre: 34400:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790740428/real 1790740428] req@0000000061964f7f x1877727329156992/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790740444 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1124.137335] LustreError: 35082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790740446 with bad export cookie 9314319877840407903 [ 1124.139520] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1124.152673] LustreError: 35082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1124.516865] Lustre: server umount lustre-MDT0001 complete [ 1137.586333] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 1140.537402] Key type lgssc unregistered [ 1140.774763] LNet: 38657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1140.780906] LNetError: 38657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1140.826867] LNet: Removed LNI 192.168.202.151@tcp [ 1141.859613] Key type .llcrypt unregistered [ 1141.862272] Key type ._llcrypt unregistered [ 1169.964096] Key type ._llcrypt registered [ 1169.971617] Key type .llcrypt registered [ 1170.161792] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1172.230552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1172.255487] alg: No test for adler32 (adler32-zlib) [ 1173.352395] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 1173.564826] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 1175.217654] Key type lgssc registered [ 1177.567751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1187.203257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1187.217116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1188.591542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1193.444501] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1198.122024] LustreError: 40094:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1198.154552] LustreError: 40094:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1202.986432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1203.172391] LustreError: 40093:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1203.486739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1208.112903] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1216.416754] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1221.745852] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1228.846992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1229.344377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1233.461774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1236.461751] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1246.600815] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1253.858090] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1253.873394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1253.885580] Lustre: Skipped 2 previous similar messages [ 1253.906125] Lustre: Skipped 2 previous similar messages [ 1258.980584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1258.990460] Lustre: Skipped 3 previous similar messages [ 1259.766507] Lustre: server umount lustre-MDT0000 complete [ 1264.114922] LustreError: 41933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1264.141204] LustreError: 41933:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1264.400375] LustreError: 41936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790740586 with bad export cookie 13830025607348389660 [ 1264.446792] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1264.630534] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.51@tcp (stopping) [ 1264.647727] Lustre: Skipped 1 previous similar message [ 1269.218264] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1269.220303] LustreError: 40088:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1269.263379] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1269.281714] LustreError: 40088:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1271.172470] Lustre: server umount lustre-MDT0001 complete [ 1294.356989] Lustre: server umount lustre-OST0000 complete [ 1314.264379] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 1317.239396] Key type lgssc unregistered [ 1317.576754] LNet: 43953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1317.597749] LNetError: 43953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1317.618572] LNet: Removed LNI 192.168.202.151@tcp [ 1319.042972] Key type .llcrypt unregistered [ 1319.050829] Key type ._llcrypt unregistered [ 1350.195268] Key type ._llcrypt registered [ 1350.196842] Key type .llcrypt registered [ 1350.318178] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1351.929241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1351.999654] alg: No test for adler32 (adler32-zlib) [ 1353.342310] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 1353.677621] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 1355.343243] Key type lgssc registered [ 1356.865889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1365.347801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1365.377280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1366.810829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1371.220729] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1376.235455] LustreError: 45387:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1376.257760] LustreError: 45387:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1379.811766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1380.253628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1384.657787] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1391.948661] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1397.675815] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1404.878379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1405.216655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1411.403274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1411.781758] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1419.670767] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1426.404559] Lustre: lustre-OST0000: Not available for connect from 192.168.202.51@tcp (stopping) [ 1426.914322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1426.916054] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1426.920522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1426.933145] Lustre: Skipped 1 previous similar message [ 1431.364640] Lustre: server umount lustre-OST0000 complete [ 1451.495557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1451.522351] Lustre: Skipped 1 previous similar message [ 1451.522544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1451.538725] Lustre: Skipped 2 previous similar messages [ 1456.611794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.620256] Lustre: Skipped 1 previous similar message [ 1461.728515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1461.740807] Lustre: Skipped 1 previous similar message [ 1464.287136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1464.486588] Lustre: server umount lustre-MDT0000 complete [ 1466.864814] LustreError: 45386:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1466.876960] LustreError: 45386:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1468.270365] LustreError: 45367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790740790 with bad export cookie 18190721545822665341 [ 1468.276090] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1468.288356] LustreError: 45367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1474.720457] Lustre: server umount lustre-MDT0001 complete [ 1489.277280] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 1492.788439] Key type lgssc unregistered [ 1493.087769] LNet: 49097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1493.103671] LNetError: 49097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1494.119357] LNet: Removed LNI 192.168.202.151@tcp [ 1495.134473] Key type .llcrypt unregistered [ 1495.135766] Key type ._llcrypt unregistered [ 1520.680247] Key type ._llcrypt registered [ 1520.682441] Key type .llcrypt registered [ 1520.780566] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1521.748332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1521.884090] alg: No test for adler32 (adler32-zlib) [ 1522.951822] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 1523.212509] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 1524.911376] Key type lgssc registered [ 1526.136676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1534.355969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1534.393865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1536.104191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1540.949434] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1545.699628] LustreError: 50531:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1545.714697] LustreError: 50531:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1550.439171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1550.878543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1556.158198] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1564.163542] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1570.979453] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1578.643664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1578.971729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1581.066162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1585.925781] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1594.174499] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1599.502814] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1603.606791] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1616.329754] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-30 0h02m15s [ 1618.163534] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1620.962391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1620.963952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1620.984233] Lustre: Skipped 2 previous similar messages [ 1621.011153] Lustre: Skipped 2 previous similar messages [ 1621.470638] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.51@tcp (stopping) [ 1622.525590] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.51@tcp (stopping) [ 1622.532746] Lustre: Skipped 3 previous similar messages [ 1627.074337] Lustre: server umount lustre-MDT0000 complete [ 1627.581869] LustreError: 50528:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1631.839983] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1633.733345] LustreError: 50528:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1633.766541] LustreError: 50528:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1639.890544] LustreError: 50526:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1639.929126] LustreError: 50526:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1641.049613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1641.161473] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1641.431726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1641.498584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1642.485867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1647.251669] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1651.416375] Lustre: DEBUG MARKER: Wait for df (36936) ... [ 1653.823846] Lustre: DEBUG MARKER: done [ 1661.924288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1661.926193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1661.938925] LustreError: Skipped 1 previous similar message [ 1661.975565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1661.992794] Lustre: Skipped 1 previous similar message [ 1668.067437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1668.085491] Lustre: Skipped 3 previous similar messages [ 1672.165433] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1672.520403] Lustre: server umount lustre-OST0000 complete [ 1678.304569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1678.306200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1678.333219] Lustre: Skipped 1 previous similar message [ 1678.335182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1678.359509] Lustre: Skipped 1 previous similar message [ 1683.342435] Lustre: server umount lustre-MDT0000 complete [ 1685.479153] LustreError: 50527:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1685.511810] LustreError: 50527:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1687.181551] LustreError: 50513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790741009 with bad export cookie 6277013970287375890 [ 1687.183148] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1687.192833] LustreError: 50513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1687.532705] Lustre: server umount lustre-MDT0001 complete [ 1699.459700] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 1702.588921] Key type lgssc unregistered [ 1702.891444] LNet: 55473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.902184] LNetError: 55473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.916434] LNet: Removed LNI 192.168.202.151@tcp [ 1703.756360] Key type .llcrypt unregistered [ 1703.760493] Key type ._llcrypt unregistered [ 1750.325824] Key type ._llcrypt registered [ 1750.329893] Key type .llcrypt registered [ 1750.564365] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1752.937914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1753.023922] alg: No test for adler32 (adler32-zlib) [ 1754.733212] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 1755.280775] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 1757.079754] Key type lgssc registered [ 1759.241724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1770.200750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1770.220849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1771.499555] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1771.574939] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1771.807285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1777.331186] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1787.218901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1787.341341] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1787.383977] Lustre: 57698:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1787.740237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1792.578847] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1800.559968] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1806.892447] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1814.658834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1815.011796] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1815.347259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1822.762776] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1825.023319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1832.110268] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1839.293391] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-30 0h05m58s [ 1841.024140] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1848.933160] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1852.047274] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1852.147743] LustreError: 60569:0:(mdt_handler.c:2345:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1872.426570] Lustre: lustre-MDT0000: Client c7b1445e-88f5-4ed9-8a41-4d19791b5825 (at 192.168.202.51@tcp) reconnecting [ 1892.167232] LustreError: 60569:0:(mdt_handler.c:2345:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1895.796925] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1898.621632] Lustre: DEBUG MARKER: done [ 1907.175732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1907.176619] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1907.209553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1907.233606] LustreError: Skipped 1 previous similar message [ 1912.296763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1912.300303] Lustre: Skipped 1 previous similar message [ 1912.791895] Lustre: server umount lustre-OST0000 complete [ 1917.409747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1917.445473] Lustre: Skipped 1 previous similar message [ 1917.452883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1917.464899] Lustre: Skipped 2 previous similar messages [ 1922.893964] Lustre: server umount lustre-MDT0000 complete [ 1926.513176] LustreError: 57689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790741248 with bad export cookie 8109620779435734367 [ 1926.514878] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1926.532046] LustreError: 57689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1926.833065] Lustre: server umount lustre-MDT0001 complete [ 1940.839685] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 1943.484195] Key type lgssc unregistered [ 1944.022567] LNet: 62037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1944.037619] LNetError: 62037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1944.052622] LNet: Removed LNI 192.168.202.151@tcp [ 1944.989687] Key type .llcrypt unregistered [ 1944.998972] Key type ._llcrypt unregistered [ 1989.168427] Key type ._llcrypt registered [ 1989.171565] Key type .llcrypt registered [ 1989.263505] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1990.378320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1990.566531] alg: No test for adler32 (adler32-zlib) [ 1991.915569] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 1992.235020] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 1993.951259] Key type lgssc registered [ 1995.355245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1999.638053] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1999.670148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2001.101342] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2001.136788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2001.400530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2002.430478] Lustre: Failing over lustre-MDT0000 [ 2002.639648] Lustre: server umount lustre-MDT0000 complete [ 2008.531632] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 00:08:48 (1790741328) [ 2014.818281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2015.248465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2019.837628] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2030.137772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2030.327281] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2030.357290] Lustre: 64852:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2030.672084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2035.185445] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2044.107958] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2050.787462] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2060.068506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2060.375195] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2060.693991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2067.844771] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2068.620126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2075.346638] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2081.760355] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2089.448642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2089.466168] Lustre: Skipped 1 previous similar message [ 2089.478782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2089.500245] Lustre: Skipped 1 previous similar message [ 2094.563181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2094.579675] Lustre: Skipped 2 previous similar messages [ 2095.894584] Lustre: server umount lustre-MDT0000 complete [ 2099.319195] LustreError: 64842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790741421 with bad export cookie 6730069784670124782 [ 2099.327162] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2099.329336] LustreError: 64842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2099.600991] Lustre: server umount lustre-MDT0001 complete [ 2103.156911] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2107.460907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2116.064322] Lustre: 63766:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790741421/real 1790741421] req@00000000d7152e76 x1877728364110464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790741437 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2116.097707] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2116.111192] Lustre: Skipped 1 previous similar message [ 2116.645315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2120.223488] Lustre: 63766:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790741426/real 1790741426] req@000000001b590973 x1877728364110592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790741442 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2124.735801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2124.882062] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2125.794395] Lustre: 63763:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790741431/real 1790741431] req@00000000eb441ba4 x1877728364110848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790741447 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2130.013270] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2130.847146] Lustre: 63764:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790741436/real 1790741436] req@000000001fb75560 x1877728364110976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790741452 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2131.950963] LustreError: 66700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2131.977272] LustreError: 66700:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2137.068481] LustreError: 68621:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2137.093597] LustreError: 68621:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2139.325750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2139.666273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2144.382588] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2144.759074] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2145.844111] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2152.238483] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2158.095124] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2169.318833] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2169.329884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2169.329970] Lustre: Skipped 1 previous similar message [ 2169.344137] Lustre: Skipped 2 previous similar messages [ 2172.357352] Lustre: server umount lustre-MDT0000 complete [ 2174.437917] LustreError: 70062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2174.474227] LustreError: 70062:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2175.998197] LustreError: 65596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790741497 with bad export cookie 6730069784670127057 [ 2175.999023] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2176.009571] LustreError: 65596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2176.373415] Lustre: server umount lustre-MDT0001 complete [ 2182.591336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2192.323519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2195.938563] Lustre: 63763:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790741501/real 1790741501] req@000000009e30e3a3 x1877728364139264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790741517 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2195.979205] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2196.001221] Lustre: Skipped 1 previous similar message [ 2201.534354] LustreError: 66701:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2201.630063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2201.689481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2205.481079] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2205.682651] LustreError: 69358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2206.689747] Lustre: 63764:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790741512/real 1790741512] req@0000000094c32900 x1877728364139904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790741528 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2206.743606] Lustre: 63764:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2213.961584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2219.506339] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2219.633601] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2224.676720] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2228.289409] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2234.857761] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2242.776229] Lustre: server umount lustre-MDT0000 complete [ 2245.097395] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2245.099367] LustreError: 66700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2245.120286] Lustre: Skipped 1 previous similar message [ 2245.128875] LustreError: 66700:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2246.912280] LustreError: 64844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790741568 with bad export cookie 6730069784670128436 [ 2246.925962] LustreError: 64844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2246.931549] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2247.396710] Lustre: server umount lustre-MDT0001 complete [ 2253.932584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2257.476972] Lustre: DEBUG MARKER: files compared the same [ 2269.900837] Lustre: server umount lustre-OST0000 complete [ 2286.555812] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 2290.106771] Key type lgssc unregistered [ 2290.431643] LNet: 74581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2290.436747] LNetError: 74581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2290.459247] LNet: Removed LNI 192.168.202.151@tcp [ 2291.669593] Key type .llcrypt unregistered [ 2291.671750] Key type ._llcrypt unregistered [ 2316.698668] Key type ._llcrypt registered [ 2316.701749] Key type .llcrypt registered [ 2316.819351] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 2317.830833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2317.861824] alg: No test for adler32 (adler32-zlib) [ 2319.111564] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 2319.375805] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 2321.120111] Key type lgssc registered [ 2322.591437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2330.039942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2330.060835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2331.615943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2336.012623] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug +malloc all [ 2341.861817] LustreError: 76034:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2341.877959] LustreError: 76034:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2344.809721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2345.115975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2349.438342] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug +malloc all [ 2358.105337] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2364.369502] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2370.726462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2371.128112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2376.231570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2377.952657] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug +malloc all [ 2385.830515] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2399.721535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2399.728276] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2399.751383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2399.770270] Lustre: Skipped 1 previous similar message [ 2404.054072] Lustre: server umount lustre-OST0000 complete [ 2409.966824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2409.974519] Lustre: Skipped 1 previous similar message [ 2409.978757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2409.985925] Lustre: Skipped 1 previous similar message [ 2415.086553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2415.098751] Lustre: Skipped 1 previous similar message [ 2419.168418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2419.178179] Lustre: Skipped 1 previous similar message [ 2422.240083] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2422.493660] Lustre: server umount lustre-MDT0000 complete [ 2424.289385] LustreError: 76028:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2424.313259] LustreError: 76028:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2425.984188] LustreError: 76014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790741747 with bad export cookie 10447109576521556297 [ 2425.987387] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2425.993443] LustreError: 76014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2426.231285] Lustre: server umount lustre-MDT0001 complete [ 2436.982766] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 2440.049049] Key type lgssc unregistered [ 2440.346780] LNet: 79569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2440.355023] LNetError: 79569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2440.379324] LNet: Removed LNI 192.168.202.151@tcp [ 2441.323458] Key type .llcrypt unregistered [ 2441.327847] Key type ._llcrypt unregistered [ 2463.531194] Key type ._llcrypt registered [ 2463.534302] Key type .llcrypt registered [ 2463.689113] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 2464.767072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2464.866062] alg: No test for adler32 (adler32-zlib) [ 2466.001927] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 2466.325240] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 2467.999197] Key type lgssc registered [ 2469.103455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2476.579823] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2476.593952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2492.063708] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2497.183616] LustreError: 80955:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.151@tcp: failed processing log, type 4: rc = -110 [ 2523.807278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2530.612707] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2538.437561] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2546.468692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2546.610599] Lustre: *** cfs_fail_loc=706, val=0*** [ 2547.002657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2547.086304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2551.381104] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2552.290788] LustreError: 80980:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.318359] LustreError: 80980:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2557.409373] LustreError: 80981:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.457894] LustreError: 82040:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.473292] LustreError: 82040:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2559.767670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2560.326541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2564.887750] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2572.587907] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2580.038627] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2586.081566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2586.090274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2586.094613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2586.102672] Lustre: Skipped 1 previous similar message [ 2589.723916] Lustre: server umount lustre-OST0000 complete [ 2596.321429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2596.328258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2596.336104] Lustre: Skipped 2 previous similar messages [ 2596.352207] Lustre: Skipped 1 previous similar message [ 2600.581352] Lustre: server umount lustre-MDT0000 complete [ 2601.440750] LustreError: 82046:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.460121] LustreError: 82046:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2605.445920] LustreError: 80963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790741927 with bad export cookie 12828895304201312888 [ 2605.450633] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2605.469379] LustreError: 80963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2605.798949] Lustre: server umount lustre-MDT0001 complete [ 2622.687978] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 2626.098128] Key type lgssc unregistered [ 2626.455857] LNet: 84595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2626.467724] LNetError: 84595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2626.502688] LNet: Removed LNI 192.168.202.151@tcp [ 2627.641230] Key type .llcrypt unregistered [ 2627.644254] Key type ._llcrypt unregistered [ 2654.681608] Key type ._llcrypt registered [ 2654.683957] Key type .llcrypt registered [ 2654.813985] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 2655.800465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2655.946858] alg: No test for adler32 (adler32-zlib) [ 2656.994294] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 2657.328599] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 2659.079346] Key type lgssc registered [ 2660.303271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2668.649791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2668.671749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2674.499441] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2682.868731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2683.630117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2687.918768] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2688.993091] LustreError: 86715:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2689.004861] LustreError: 86715:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2694.117751] LustreError: 86719:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2694.130456] LustreError: 86719:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2696.249456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2696.535335] LustreError: 86718:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2696.558697] LustreError: 86718:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2696.579250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2701.795551] LustreError: 86719:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2701.811951] LustreError: 86719:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2703.741629] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2706.914634] LustreError: 86713:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2706.944684] LustreError: 86713:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2713.340127] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2713.346539] Lustre: MGS: Client a129ea1e-52ce-47d8-b24c-d8a31bb9e29a (at 0@lo) reconnecting [ 2713.363650] LustreError: 86729:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2713.631485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2713.725104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2719.715748] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2735.585466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2735.596881] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2735.627908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2739.172724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2739.188169] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2739.208896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.527589] Lustre: server umount lustre-OST0000 complete [ 2749.411779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2749.415424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2749.417710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2749.417716] Lustre: Skipped 1 previous similar message [ 2749.448590] Lustre: Skipped 1 previous similar message [ 2751.849668] Lustre: server umount lustre-MDT0000 complete [ 2754.529644] LustreError: 88259:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2754.544907] LustreError: 88259:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2755.747724] Lustre: server umount lustre-MDT0001 complete [ 2759.403371] Lustre: server umount MGS complete [ 2770.520292] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 00:21:30 (1790742090) [ 2800.284905] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 2810.642730] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 2845.705417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2851.443495] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2862.796718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2863.184302] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2863.295547] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2863.427127] Lustre: lustre-MDT0001: new disk, initializing [ 2863.579053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2863.633352] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2863.648209] Lustre: 93907:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2863.659137] LustreError: 93907:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2869.675172] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2881.030663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2881.397155] Lustre: lustre-OST0000: new disk, initializing [ 2881.406574] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2881.420156] Lustre: 94673:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2881.540174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2887.085492] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2891.302384] Lustre: 95149:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000d5008ac2 x1877729061414016/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790742229 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2895.988786] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2902.376640] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2902.385737] Lustre: MGS: Client 8387bc6f-7559-45f9-9d99-161e63441786 (at 0@lo) reconnecting [ 2902.428425] Lustre: 95719:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2902.444807] LustreError: 93893:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2902.473414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2902.588637] Lustre: lustre-MDT0000: new disk, initializing [ 2902.782274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2902.855878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2902.966551] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2903.904104] Lustre: 95743:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000cc8b7896 x1877729061420416/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1790742241 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2907.002778] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2911.713535] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2913.069466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2913.087318] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2913.160484] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2937.835857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2937.843975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2937.848085] Lustre: Skipped 1 previous similar message [ 2937.868262] Lustre: Skipped 2 previous similar messages [ 2942.952154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2942.961607] Lustre: Skipped 1 previous similar message [ 2943.301735] Lustre: server umount lustre-OST0000 complete [ 2948.064724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2952.860268] Lustre: server umount lustre-MDT0000 complete [ 2953.185523] LustreError: 93882:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2953.201184] LustreError: 93882:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2963.009454] Lustre: server umount lustre-MDT0001 complete [ 2966.202447] Lustre: server umount MGS complete [ 2975.462874] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 00:24:55 (1790742295) [ 2992.051441] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 2994.084158] Key type lgssc unregistered [ 2994.436225] LNet: 98238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2994.451722] LNetError: 98238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2994.480419] LNet: Removed LNI 192.168.202.151@tcp [ 2995.256578] Key type .llcrypt unregistered [ 2995.259230] Key type ._llcrypt unregistered [ 3011.946676] Key type ._llcrypt registered [ 3011.953340] Key type .llcrypt registered [ 3012.084865] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3013.099248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3013.207879] alg: No test for adler32 (adler32-zlib) [ 3014.509790] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 3014.847398] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 3016.567442] Key type lgssc registered [ 3018.199299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3028.364762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3028.390703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3029.954146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3035.498958] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3039.204790] LustreError: 99493:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3039.254430] LustreError: 99493:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3043.886364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3044.198933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3049.187311] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3057.902779] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3063.589685] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3070.127530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3070.519614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3074.637996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3078.328514] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3087.199432] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3100.129326] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3100.134616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3100.144705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3100.643965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3100.654149] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3100.670405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3105.786887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3105.804139] Lustre: Skipped 1 previous similar message [ 3106.586808] Lustre: server umount lustre-OST0000 complete [ 3116.001236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3116.001718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3116.029716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3121.120958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3121.123860] Lustre: Skipped 1 previous similar message [ 3125.215146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3125.576935] Lustre: server umount lustre-MDT0000 complete [ 3126.242250] LustreError: 99492:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3126.276046] LustreError: 99492:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3129.886043] LustreError: 99474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790742451 with bad export cookie 1951337619217357123 [ 3129.895168] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3129.897834] LustreError: 99474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3130.282609] Lustre: server umount lustre-MDT0001 complete [ 3144.013573] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 3147.731742] Key type lgssc unregistered [ 3148.194938] LNet: 103050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3148.215311] LNetError: 103050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3149.287505] LNet: Removed LNI 192.168.202.151@tcp [ 3150.707424] Key type .llcrypt unregistered [ 3150.709411] Key type ._llcrypt unregistered [ 3167.609177] Key type ._llcrypt registered [ 3167.612776] Key type .llcrypt registered [ 3167.718848] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3168.749397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3168.970380] alg: No test for adler32 (adler32-zlib) [ 3170.125841] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 3170.361706] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 3172.031209] Key type lgssc registered [ 3173.286402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3179.610414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3179.617640] LustreError: 104151:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3179.624341] Lustre: Skipped 1 previous similar message [ 3179.626808] LustreError: 104150:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3179.631710] LustreError: 104151:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4990 [ 3179.640790] LustreError: 104151:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3179.658500] LustreError: 104151:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3179.664666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3181.136434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3188.591221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3188.871983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3193.540653] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3198.968270] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing lsmod [ 3201.640938] LustreError: 105352:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3201.664330] LustreError: 105357:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3201.668421] LustreError: 105352:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4978 [ 3201.680564] LustreError: 105352:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3201.696264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3201.697055] LustreError: 105352:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3202.006933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3204.807193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3204.824968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3204.832828] Lustre: Skipped 1 previous similar message [ 3209.704843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3209.720053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3209.741085] Lustre: Skipped 1 previous similar message [ 3209.746533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3211.473434] Lustre: server umount lustre-MDT0000 complete [ 3213.282415] LustreError: 104189:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3213.296088] LustreError: 104189:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3215.225202] LustreError: 104169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790742537 with bad export cookie 15426021322121812107 [ 3215.237989] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3215.242156] LustreError: 104169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3215.356241] LustreError: 104619:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002d769011 x1877729599311104/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3215.489101] Lustre: server umount lustre-MDT0001 complete [ 3229.113348] Lustre: server umount lustre-OST0000 complete [ 3237.282976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3237.786936] LustreError: 106420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3237.878994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3242.390264] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3242.979801] LustreError: 106421:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3248.128585] LustreError: 106420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3250.888229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3251.242346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3255.861483] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3263.624744] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3269.202848] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3276.111775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3276.462953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3283.252193] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3283.646932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3290.087607] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3298.285270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3298.296463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3298.315094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3298.323966] Lustre: Skipped 3 previous similar messages [ 3299.316699] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3299.715512] Lustre: server umount lustre-OST0000 complete [ 3304.423349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3304.429251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3304.444477] Lustre: Skipped 2 previous similar messages [ 3304.462628] Lustre: Skipped 3 previous similar messages [ 3309.540332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3309.543088] Lustre: Skipped 1 previous similar message [ 3310.036284] Lustre: server umount lustre-MDT0000 complete [ 3313.686213] LustreError: 107156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790742635 with bad export cookie 15426021322121812793 [ 3313.687644] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3313.702065] LustreError: 107156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3313.999837] Lustre: server umount lustre-MDT0001 complete [ 3326.014491] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 3328.734826] Key type lgssc unregistered [ 3329.108590] LNet: 109949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3329.115883] LNetError: 109949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3329.132755] LNet: Removed LNI 192.168.202.151@tcp [ 3330.034398] Key type .llcrypt unregistered [ 3330.043630] Key type ._llcrypt unregistered [ 3352.467427] Key type ._llcrypt registered [ 3352.469440] Key type .llcrypt registered [ 3352.564717] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3353.879624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3353.930941] alg: No test for adler32 (adler32-zlib) [ 3354.920156] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 3355.186439] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 3356.863176] Key type lgssc registered [ 3358.274895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3366.216470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3366.236967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3367.651495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3372.565123] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3377.122716] LustreError: 111379:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3377.149748] LustreError: 111379:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3381.822858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3382.290256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3386.477554] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3394.591936] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3401.178416] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3409.056270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3409.508970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3412.609484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3416.692574] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3425.246414] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3431.356816] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3433.643545] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3438.562272] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3438.573436] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3438.578911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3438.583727] LustreError: Skipped 1 previous similar message [ 3438.614314] Lustre: Skipped 1 previous similar message [ 3443.708867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3443.717849] Lustre: Skipped 1 previous similar message [ 3444.208897] Lustre: server umount lustre-OST0000 complete [ 3448.801584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3448.803585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3448.816467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3448.827050] Lustre: Skipped 1 previous similar message [ 3448.842646] Lustre: Skipped 2 previous similar messages [ 3453.924864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3453.941115] Lustre: Skipped 1 previous similar message [ 3454.737183] Lustre: server umount lustre-MDT0000 complete [ 3459.042109] LustreError: 111379:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3459.074561] LustreError: 111379:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3459.459015] LustreError: 111361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790742781 with bad export cookie 14137005305673536973 [ 3459.461795] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3459.471331] LustreError: 111361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3460.030907] Lustre: server umount lustre-MDT0001 complete [ 3473.032400] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 3476.077920] Key type lgssc unregistered [ 3476.351691] LNet: 115018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3476.359487] LNetError: 115018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3476.384068] LNet: Removed LNI 192.168.202.151@tcp [ 3477.342817] Key type .llcrypt unregistered [ 3477.346043] Key type ._llcrypt unregistered [ 3493.755695] Key type ._llcrypt registered [ 3493.759930] Key type .llcrypt registered [ 3493.858378] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3495.194303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3495.219528] alg: No test for adler32 (adler32-zlib) [ 3496.357371] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 3496.591544] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 3498.255170] Key type lgssc registered [ 3499.579969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3507.427117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3507.448668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3508.946258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3512.978680] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3518.442457] LustreError: 116249:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3518.471665] LustreError: 116249:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3520.708772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3520.999822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3525.046231] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3531.517576] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3536.427969] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3543.636707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3544.119165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3550.413755] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3553.331089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3557.535445] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3565.556424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3565.566947] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3565.611291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3567.084453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3567.087064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3567.108331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3568.612501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3568.619765] Lustre: Skipped 1 previous similar message [ 3571.547450] Lustre: server umount lustre-OST0000 complete [ 3577.313397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3577.325877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3577.333827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3582.656755] Lustre: server umount lustre-MDT0000 complete [ 3583.970186] LustreError: 116248:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3584.015333] LustreError: 116248:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3587.565877] LustreError: 116983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790742909 with bad export cookie 13898364380901967259 [ 3587.569698] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3587.590504] LustreError: 116983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3588.317435] Lustre: server umount lustre-MDT0001 complete [ 3600.815124] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 3603.714292] Key type lgssc unregistered [ 3604.010239] LNet: 119783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3604.016553] LNetError: 119783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3604.030584] LNet: Removed LNI 192.168.202.151@tcp [ 3605.009085] Key type .llcrypt unregistered [ 3605.012703] Key type ._llcrypt unregistered [ 3629.491472] Key type ._llcrypt registered [ 3629.493972] Key type .llcrypt registered [ 3629.600765] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3630.802693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3630.834501] alg: No test for adler32 (adler32-zlib) [ 3631.884297] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 3632.066609] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 3633.721715] Key type lgssc registered [ 3634.817710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3642.259678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3642.280924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3643.736617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3649.619316] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3654.114166] LustreError: 121214:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3654.130406] LustreError: 121214:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3657.755815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3658.217939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3663.154225] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3670.849304] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3676.112599] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3681.910846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3682.134497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3688.302383] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3689.333741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3695.735533] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3703.694972] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3706.797165] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3706.801872] Lustre: Skipped 2 previous similar messages [ 3709.564103] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3709.609334] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3710.584093] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3710.589416] Lustre: Skipped 2 previous similar messages [ 3715.150538] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3715.155827] Lustre: Skipped 1 previous similar message [ 3719.981512] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.51@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3719.987894] Lustre: Skipped 2 previous similar messages [ 3720.879084] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.51@tcp 192.168.0.[2,10]@tcp [ 3733.185103] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.51/24@tcp in log lustre-MDT0000 [ 3733.194909] Lustre: Skipped 2 previous similar messages [ 3743.331742] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.51/24@tcp [ 3743.344515] Lustre: Skipped 1 previous similar message [ 3749.856442] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3749.869468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3749.911402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3750.882982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3750.883896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3750.884584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3750.893828] Lustre: Skipped 1 previous similar message [ 3752.978167] Lustre: server umount lustre-OST0000 complete [ 3760.616962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3760.625601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3760.651712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3762.895177] Lustre: server umount lustre-MDT0000 complete [ 3766.244920] LustreError: 121208:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3766.260588] LustreError: 121208:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3766.575456] LustreError: 121193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790743088 with bad export cookie 14402879086391780681 [ 3766.580492] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3766.584557] LustreError: 121193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3766.845720] Lustre: server umount lustre-MDT0001 complete [ 3778.154276] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 3780.407185] Key type lgssc unregistered [ 3780.775830] LNet: 126203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3780.811732] LNetError: 126203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3781.861789] LNet: Removed LNI 192.168.202.151@tcp [ 3782.810607] Key type .llcrypt unregistered [ 3782.815744] Key type ._llcrypt unregistered [ 3807.739981] Key type ._llcrypt registered [ 3807.742541] Key type .llcrypt registered [ 3807.814918] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3808.527355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3808.564377] alg: No test for adler32 (adler32-zlib) [ 3809.672827] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 3809.909322] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 3811.575245] Key type lgssc registered [ 3812.803704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3819.143320] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3819.178982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3820.581604] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3820.600284] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.51/24@tcp [ 3820.618546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3824.185901] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3830.759799] LustreError: 127760:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3830.777949] LustreError: 127760:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3832.879554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3833.256052] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3833.287904] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.51/24@tcp [ 3833.323842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3838.076148] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3845.010795] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3850.577094] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3857.023145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3857.320983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3863.048721] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3865.534588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3870.897903] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3884.001379] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3884.009284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3884.034027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3886.050608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3886.052577] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3886.053535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3886.065826] Lustre: Skipped 1 previous similar message [ 3887.263890] Lustre: server umount lustre-OST0000 complete [ 3891.175477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3891.180256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3891.187969] Lustre: Skipped 1 previous similar message [ 3891.201360] Lustre: Skipped 1 previous similar message [ 3896.289515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3896.298731] Lustre: Skipped 1 previous similar message [ 3897.329150] Lustre: server umount lustre-MDT0000 complete [ 3900.667671] LustreError: 127742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790743222 with bad export cookie 7870991531407061287 [ 3900.675371] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3900.678367] LustreError: 127742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3900.928599] Lustre: server umount lustre-MDT0001 complete [ 3912.282356] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 3914.841369] Key type lgssc unregistered [ 3915.152201] LNet: 131347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3915.160608] LNetError: 131347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3916.199779] LNet: Removed LNI 192.168.202.151@tcp [ 3917.206399] Key type .llcrypt unregistered [ 3917.208380] Key type ._llcrypt unregistered [ 3940.910715] Key type ._llcrypt registered [ 3940.912975] Key type .llcrypt registered [ 3940.999880] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 3953.683236] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3954.753281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3954.763941] alg: No test for adler32 (adler32-zlib) [ 3955.813435] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 3956.032045] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 3957.703320] Key type lgssc registered [ 3958.536076] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3990.363935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3990.380961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3991.782344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3991.825583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3991.894580] Lustre: lustre-MDT0000: new disk, initializing [ 3991.977606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3991.997634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3995.546430] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4006.080182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4006.177574] Lustre: 134741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4006.215809] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4006.218214] Lustre: Skipped 1 previous similar message [ 4006.270490] Lustre: lustre-MDT0001: new disk, initializing [ 4006.332607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4006.363061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4006.392408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4009.865994] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4020.981248] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4027.653546] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4034.970984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4035.249908] Lustre: lustre-OST0000: new disk, initializing [ 4035.254766] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4035.264828] Lustre: 136709:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4035.326515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4041.306863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4041.316830] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4041.430211] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4041.652397] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4050.841697] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4054.993501] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4055.130459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4059.858994] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4060.025887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4062.187571] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4062.191910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4062.203551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4062.206573] Lustre: Skipped 1 previous similar message [ 4062.232471] Lustre: Skipped 1 previous similar message [ 4067.304599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4067.318282] Lustre: Skipped 1 previous similar message [ 4067.671118] Lustre: server umount lustre-OST0000 complete [ 4072.419199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4072.437651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4072.448508] Lustre: Skipped 1 previous similar message [ 4077.478553] Lustre: server umount lustre-MDT0000 complete [ 4081.097737] LustreError: 134732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790743402 with bad export cookie 11594672163530951813 [ 4081.103144] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4081.125958] LustreError: 134732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4081.779718] Lustre: server umount lustre-MDT0001 complete [ 4091.834444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4092.520134] LustreError: 138925:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4092.547992] LustreError: 138925:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4092.666374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4098.024254] LustreError: 138926:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4098.129773] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4102.112618] LustreError: 138925:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4107.232660] LustreError: 138926:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4107.329720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4107.719076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4112.830378] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4121.176580] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4127.680719] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4133.728720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4134.004599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4140.115645] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4146.701487] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4151.263940] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4151.548628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4156.612766] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4156.761627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4165.170082] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4165.336410] Lustre: lustre-OST0001: new disk, initializing [ 4165.339803] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4165.348174] Lustre: 142298:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4165.418195] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4170.987213] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4171.918593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4171.934604] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4172.045936] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4178.921258] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4179.156861] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4183.463910] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4183.709668] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4188.892792] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ce1d0bc4000.ost_server_uuid 50 [ 4190.445825] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8ce1d0bc4000.ost_server_uuid in IDLE state after 0 sec [ 4202.976452] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4202.989206] LustreError: Skipped 1 previous similar message [ 4202.999452] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4203.016576] Lustre: Skipped 1 previous similar message [ 4203.049089] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4204.713992] Lustre: server umount lustre-OST0001 complete [ 4208.101334] LustreError: 140746:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4208.123852] LustreError: 140746:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4209.131200] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4209.150391] LustreError: Skipped 1 previous similar message [ 4209.155000] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4209.180795] Lustre: Skipped 1 previous similar message [ 4209.186769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4209.189556] Lustre: Skipped 1 previous similar message [ 4214.773705] Lustre: server umount lustre-OST0000 complete [ 4218.337365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4218.347761] Lustre: Skipped 1 previous similar message [ 4218.351570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4218.354243] Lustre: Skipped 3 previous similar messages [ 4224.466225] Lustre: server umount lustre-MDT0000 complete [ 4227.579893] LustreError: 138906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790743549 with bad export cookie 11594672163530952695 [ 4227.591651] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4227.597453] LustreError: 138906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4227.888771] Lustre: server umount lustre-MDT0001 complete [ 4247.373220] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 4250.435073] Key type lgssc unregistered [ 4250.723803] LNet: 145303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4250.730109] LNetError: 145303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4250.761256] LNet: Removed LNI 192.168.202.151@tcp [ 4251.984119] Key type .llcrypt unregistered [ 4251.987230] Key type ._llcrypt unregistered [ 4287.549633] Key type ._llcrypt registered [ 4287.551855] Key type .llcrypt registered [ 4287.659381] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 4298.876856] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 4299.701271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4299.722825] alg: No test for adler32 (adler32-zlib) [ 4300.800562] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 4301.019073] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 4302.671747] Key type lgssc registered [ 4303.491848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4330.695812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4330.714601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4331.884447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4331.906243] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4331.996204] Lustre: lustre-MDT0000: new disk, initializing [ 4332.083515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4332.098752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4335.857379] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4345.137411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4345.240879] Lustre: 149449:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4345.274783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4345.281398] Lustre: Skipped 1 previous similar message [ 4345.343453] Lustre: lustre-MDT0001: new disk, initializing [ 4345.456397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4345.490231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4345.501459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4349.878958] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4358.590215] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4364.132651] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4369.998986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4370.165243] Lustre: lustre-OST0000: new disk, initializing [ 4370.167647] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4370.171693] Lustre: 151422:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4370.240268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4376.026467] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4376.136867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4376.152841] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4376.276348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4383.873707] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4388.207414] Lustre: Failing over lustre-OST0000 [ 4388.375697] Lustre: server umount lustre-OST0000 complete [ 4406.187604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4406.239517] Lustre: 147665:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790743712/real 1790743712] req@0000000008cb40d2 x1877730784214784/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1790743728 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4406.263365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4406.653983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4406.672344] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4407.268524] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4407.732277] Lustre: 147667:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790743713/real 1790743713] req@0000000072a37000 x1877730784215680/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790743729 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4407.757066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4412.391675] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4412.833441] Lustre: 147666:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790743718/real 1790743718] req@0000000060a51671 x1877730784216960/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790743734 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4412.862670] Lustre: 147666:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4416.653626] Lustre: Failing over lustre-MDT0000 [ 4416.676243] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4416.702237] Lustre: Skipped 2 previous similar messages [ 4416.708431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4416.979169] Lustre: server umount lustre-MDT0000 complete [ 4418.791902] Lustre: 147666:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790743724/real 1790743724] req@0000000049b89b3d x1877730784217984/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790743740 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4418.829967] Lustre: 147666:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4422.590411] LustreError: 149458:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4422.630493] LustreError: 149458:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4422.943408] Lustre: 147666:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790743728/real 1790743728] req@0000000038ced2cd x1877730784219008/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790743744 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4427.235456] LustreError: 150677:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4427.262840] LustreError: 150677:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4432.361785] LustreError: 149462:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4432.403414] LustreError: 149462:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4436.251323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4436.456293] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4436.682456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4436.749736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4436.801992] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4437.949171] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4441.186599] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4442.099075] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4442.121421] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4442.208814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4452.324208] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4452.325466] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4452.337087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4452.337095] Lustre: Skipped 2 previous similar messages [ 4452.374785] Lustre: Skipped 1 previous similar message [ 4454.382223] Lustre: server umount lustre-OST0000 complete [ 4462.560168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4462.564535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4462.583694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4462.591889] Lustre: Skipped 2 previous similar messages [ 4467.680159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4467.694337] Lustre: Skipped 1 previous similar message [ 4472.291091] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4472.606650] Lustre: server umount lustre-MDT0000 complete [ 4472.817646] LustreError: 150565:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4472.840585] LustreError: 150565:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4476.460346] LustreError: 149442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790743798 with bad export cookie 17395380091911406848 [ 4476.464409] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4476.475685] LustreError: 149442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4476.894583] Lustre: server umount lustre-MDT0001 complete [ 4489.168602] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 4492.344335] Key type lgssc unregistered [ 4492.724402] LNet: 155073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4492.730264] LNetError: 155073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4492.749827] LNet: Removed LNI 192.168.202.151@tcp [ 4493.575368] Key type .llcrypt unregistered [ 4493.578224] Key type ._llcrypt unregistered [ 4528.436740] Key type ._llcrypt registered [ 4528.439864] Key type .llcrypt registered [ 4528.561753] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 4529.739290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4529.865177] alg: No test for adler32 (adler32-zlib) [ 4530.852185] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 4531.083632] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 4532.807340] Key type lgssc registered [ 4534.639887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4543.851589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4543.876539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4545.486914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4550.077875] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4555.748654] LustreError: 156622:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4555.775902] LustreError: 156622:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4560.086682] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4560.868045] LustreError: 156621:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4565.778753] LustreError: 156650:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4565.792788] LustreError: 156650:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4565.801228] LustreError: 156650:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4566.400286] Lustre: server umount lustre-MDT0000 complete [ 4583.415368] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 4596.583739] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 4636.117441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4636.408805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4636.446152] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4636.552471] Lustre: lustre-MDT0000: new disk, initializing [ 4636.680460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4636.703055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4641.112470] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4650.963120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4651.148244] Lustre: 159954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4651.192925] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4651.195474] Lustre: Skipped 1 previous similar message [ 4651.264181] Lustre: lustre-MDT0001: new disk, initializing [ 4651.330628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4651.362435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4651.380458] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4655.885268] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4665.291433] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4670.640705] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4677.425836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4677.795271] Lustre: lustre-OST0000: new disk, initializing [ 4677.809381] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4677.817242] Lustre: 161923:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4677.929776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4684.921662] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4685.362200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4685.376585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4685.522624] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4695.345885] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4701.466499] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4704.036745] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 00:53:43 (1790744023) [ 4931.553851] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4931.573717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4931.578175] Lustre: Skipped 1 previous similar message [ 4931.585637] Lustre: Skipped 1 previous similar message [ 4936.678375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4936.692065] Lustre: Skipped 2 previous similar messages [ 4937.316378] Lustre: server umount lustre-MDT0000 complete [ 4941.635772] LustreError: 159945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790744263 with bad export cookie 16339067561763615442 [ 4941.637110] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4941.670698] LustreError: 159945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4941.791838] LustreError: 159962:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4941.801488] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4941.828753] Lustre: Skipped 1 previous similar message [ 4941.841748] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4946.912782] LustreError: 161200:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4946.912793] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4952.041116] LustreError: 159962:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4952.041504] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4956.127130] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4956.570422] Lustre: server umount lustre-MDT0001 complete [ 4975.072982] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4975.234360] Lustre: server umount lustre-OST0000 complete [ 4985.015296] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 4995.597580] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 5036.839275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5037.223442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5037.280865] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5037.410703] Lustre: lustre-MDT0000: new disk, initializing [ 5037.567930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5037.593721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5042.540702] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5053.767844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5053.888798] Lustre: 166588:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5053.998178] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5054.013711] Lustre: Skipped 1 previous similar message [ 5054.089951] Lustre: lustre-MDT0001: new disk, initializing [ 5054.208411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5054.265780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5054.275475] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5059.461500] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5069.629054] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5075.599490] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5082.764788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5083.051962] Lustre: lustre-OST0000: new disk, initializing [ 5083.059671] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5083.075406] Lustre: 168557:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5083.255291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5088.868548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5088.883031] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5089.055178] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5090.714722] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5100.708857] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5105.592430] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5105.840778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5111.073518] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5111.319944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5114.848370] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5114.850427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5114.862727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5114.898246] Lustre: Skipped 1 previous similar message [ 5119.911406] Lustre: server umount lustre-OST0000 complete [ 5125.089108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5130.203718] Lustre: server umount lustre-MDT0000 complete [ 5130.239078] LustreError: 167398:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5134.329103] LustreError: 166581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790744456 with bad export cookie 16339067561763774272 [ 5134.334225] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5134.351619] LustreError: 166581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5134.729992] Lustre: server umount lustre-MDT0001 complete [ 5143.725617] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:01:03 (1790744463) [ 5160.071692] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 5169.440052] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 5205.433306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5205.684529] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5205.703066] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5205.773189] Lustre: lustre-MDT0000: new disk, initializing [ 5205.905552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5205.951500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5210.814934] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5222.746334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5222.954398] Lustre: 173324:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5223.302963] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5228.747642] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5239.936939] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5245.732467] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5252.624951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5253.041838] Lustre: lustre-OST0000: new disk, initializing [ 5253.044956] Lustre: Skipped 1 previous similar message [ 5253.050192] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5253.053418] Lustre: Skipped 2 previous similar messages [ 5253.057399] Lustre: 175300:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5259.803662] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5261.931971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5261.945051] Lustre: Skipped 1 previous similar message [ 5261.963772] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5262.022053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5270.272607] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5277.666170] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5277.672849] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5277.686691] LustreError: Skipped 1 previous similar message [ 5277.692114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5277.704482] Lustre: Skipped 2 previous similar messages [ 5277.722990] Lustre: Skipped 4 previous similar messages [ 5283.091348] Lustre: server umount lustre-OST0000 complete [ 5287.905175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5287.932418] Lustre: Skipped 1 previous similar message [ 5293.575672] Lustre: server umount lustre-MDT0000 complete [ 5297.836231] LustreError: 175299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790744619 with bad export cookie 16339067561763775154 [ 5297.848457] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5298.212652] Lustre: server umount lustre-MDT0001 complete [ 5306.560502] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:03:46 (1790744626) [ 5322.040819] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 5330.649458] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 5372.019979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5372.278403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5372.344298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5372.437416] Lustre: lustre-MDT0000: new disk, initializing [ 5372.545700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5372.549226] Lustre: Skipped 2 previous similar messages [ 5372.569338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5378.187773] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5388.757510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5388.877610] Lustre: 179623:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5389.200767] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5394.266743] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5404.696215] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5410.954587] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5418.089822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5418.313613] Lustre: 181601:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5425.119104] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5426.205725] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5426.360732] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5435.650280] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5446.624538] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5446.639916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5446.673702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5446.682853] Lustre: Skipped 6 previous similar messages [ 5448.955636] Lustre: server umount lustre-OST0000 complete [ 5455.840793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5455.848142] LustreError: Skipped 1 previous similar message [ 5459.691639] Lustre: server umount lustre-MDT0000 complete [ 5462.499279] LustreError: 180851:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5462.516667] LustreError: 180851:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5463.848968] LustreError: 179614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790744785 with bad export cookie 16339067561763776232 [ 5463.862838] LustreError: 179614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5463.872172] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5464.248891] Lustre: server umount lustre-MDT0001 complete [ 5477.313844] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 5479.814799] Key type lgssc unregistered [ 5480.099163] LNet: 183456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5480.116657] LNetError: 183456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5481.192595] LNet: Removed LNI 192.168.202.151@tcp [ 5482.165636] Key type .llcrypt unregistered [ 5482.166855] Key type ._llcrypt unregistered [ 5505.535102] Key type ._llcrypt registered [ 5505.537236] Key type .llcrypt registered [ 5505.667532] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 5506.660531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5506.695184] alg: No test for adler32 (adler32-zlib) [ 5507.824484] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 5508.000732] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 5509.695677] Key type lgssc registered [ 5510.657402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5518.337890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5518.349846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5519.847494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5525.812848] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5529.121608] LustreError: 184887:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5529.132754] LustreError: 184887:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5534.177200] LustreError: 184888:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5534.718105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5535.101889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5539.828824] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5547.828576] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5553.204361] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5559.903095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5560.287424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5564.394321] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5565.508716] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5573.056649] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5600.229783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5600.230950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5600.236568] LustreError: Skipped 1 previous similar message [ 5600.244624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5600.257697] Lustre: Skipped 1 previous similar message [ 5602.501524] Lustre: server umount lustre-OST0000 complete [ 5606.880703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5606.893302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5606.904400] Lustre: Skipped 1 previous similar message [ 5606.913725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5610.467664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5610.472993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5610.497026] Lustre: Skipped 1 previous similar message [ 5612.972161] Lustre: server umount lustre-MDT0000 complete [ 5615.589908] LustreError: 184882:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5615.610820] LustreError: 184882:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5616.749080] LustreError: 184868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790744938 with bad export cookie 17979239346567569944 [ 5616.761380] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5616.763769] LustreError: 184868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5617.150898] Lustre: server umount lustre-MDT0001 complete [ 5631.286046] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 5634.667665] Key type lgssc unregistered [ 5635.003668] LNet: 188443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5635.009619] LNetError: 188443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5635.022302] LNet: Removed LNI 192.168.202.151@tcp [ 5635.832658] Key type .llcrypt unregistered [ 5635.834395] Key type ._llcrypt unregistered [ 5658.853276] Key type ._llcrypt registered [ 5658.855719] Key type .llcrypt registered [ 5658.978743] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 5660.280043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5660.364882] alg: No test for adler32 (adler32-zlib) [ 5661.413541] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 5661.622960] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 5663.343601] Key type lgssc registered [ 5664.588875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5671.981397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5672.005719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5673.539879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5678.172264] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5683.680927] LustreError: 189877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5683.705957] LustreError: 189877:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5687.581293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5688.217164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5693.389463] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5701.658936] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5707.181527] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5713.861718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5714.136344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5718.271654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5720.093210] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5727.881622] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5737.956574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5737.960170] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5737.981075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5739.903311] Lustre: server umount lustre-OST0000 complete [ 5746.582750] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8ce1c76f3000.ost_server_uuid 50 [ 5760.169736] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5772.769154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5772.773701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5772.779321] Lustre: Skipped 2 previous similar messages [ 5772.790919] Lustre: Skipped 2 previous similar messages [ 5777.889273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5777.897778] Lustre: Skipped 1 previous similar message [ 5778.495294] Lustre: server umount lustre-MDT0000 complete [ 5781.044850] LustreError: 192943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790745102 with bad export cookie 13207279372057035398 [ 5781.055818] LustreError: 192943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5781.057305] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5781.291957] Lustre: server umount lustre-MDT0001 complete [ 5788.753395] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 01:11:48 (1790745108) [ 5793.904381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5794.223353] LustreError: 193812:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5794.270512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5797.717541] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5799.403962] LustreError: 193813:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5804.519572] LustreError: 193812:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5805.754409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5806.114767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5810.621360] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5818.185171] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5823.157813] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5829.732642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5830.147867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5836.347681] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5839.348411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5842.018901] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5847.305794] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5847.416716] Lustre: lustre-OST0001: new disk, initializing [ 5847.420381] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5847.426574] Lustre: 196652:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5847.488598] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5853.624876] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5854.334369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5854.343638] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5854.471155] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5863.203953] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5866.976982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5866.985532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5867.003821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5870.064849] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5870.065012] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5871.466065] Lustre: server umount lustre-OST0000 complete [ 5874.110183] LustreError: 195633:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5877.231325] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5877.434875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5881.289640] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5881.486311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5884.349343] LustreError: 195633:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5884.373587] LustreError: 195633:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5895.649255] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5895.669072] Lustre: Skipped 1 previous similar message [ 5895.671463] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5895.671610] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5895.705982] Lustre: Skipped 4 previous similar messages [ 5900.769931] LustreError: 195631:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5900.792505] LustreError: 195631:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 5901.749911] Lustre: server umount lustre-OST0001 complete [ 5905.896423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5905.911562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5905.928136] Lustre: Skipped 3 previous similar messages [ 5911.668906] Lustre: server umount lustre-MDT0000 complete [ 5915.553671] LustreError: 193795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790745237 with bad export cookie 13207279372057036476 [ 5915.558988] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5915.562311] LustreError: 193795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5916.077080] Lustre: server umount lustre-MDT0001 complete [ 5939.600765] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 01:14:19 (1790745259) [ 5945.534557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5945.843729] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5945.888637] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5946.147559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5950.886701] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5958.679563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5958.768075] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5958.803994] Lustre: 200091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5963.292415] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5970.386857] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5976.283383] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5982.666579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5982.864886] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5988.484256] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5990.091590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5995.211281] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6001.327581] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6001.411122] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6007.767469] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6011.095889] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6015.624893] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6020.580866] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6020.595329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6020.623505] Lustre: Skipped 1 previous similar message [ 6020.635661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6020.647783] Lustre: Skipped 2 previous similar messages [ 6024.622844] Lustre: server umount lustre-OST0000 complete [ 6026.693857] LustreError: 201943:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6042.085461] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6042.091708] LustreError: Skipped 1 previous similar message [ 6046.904509] Lustre: server umount lustre-OST0001 complete [ 6056.563390] Lustre: server umount lustre-MDT0000 complete [ 6059.669567] LustreError: 200084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790745381 with bad export cookie 13207279372057037883 [ 6059.671224] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6059.681698] LustreError: 200084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6060.013129] Lustre: server umount lustre-MDT0001 complete [ 6083.188536] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 01:16:43 (1790745403) [ 6098.244709] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 6107.891988] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 6138.859929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6139.083515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6139.105705] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6139.197181] Lustre: lustre-MDT0000: new disk, initializing [ 6139.272538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6139.277714] Lustre: Skipped 3 previous similar messages [ 6139.287797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6143.630043] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6153.942985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6154.177574] Lustre: 208206:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6154.261085] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6154.264622] Lustre: Skipped 1 previous similar message [ 6154.346066] Lustre: lustre-MDT0001: new disk, initializing [ 6154.471605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6154.496401] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6159.824908] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6168.764967] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6174.028787] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6180.307515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6180.597293] Lustre: lustre-OST0000: new disk, initializing [ 6180.603513] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6180.615198] Lustre: 210170:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6186.463926] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6189.127369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6189.135214] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6189.308285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6194.687502] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6198.575159] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6198.792048] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6203.134190] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6203.348517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6210.021746] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6210.026826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6210.031980] LustreError: Skipped 2 previous similar messages [ 6210.033800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6210.033878] Lustre: Skipped 8 previous similar messages [ 6210.064216] Lustre: Skipped 6 previous similar messages [ 6219.234044] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6219.519310] Lustre: server umount lustre-OST0000 complete [ 6231.043646] Lustre: server umount lustre-MDT0000 complete [ 6234.973497] LustreError: 208196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790745556 with bad export cookie 13207279372057039668 [ 6234.984261] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6234.991148] LustreError: 208196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6235.599866] Lustre: server umount lustre-MDT0001 complete [ 6244.163151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6244.802377] LustreError: 212387:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6244.824023] LustreError: 212387:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 6250.694712] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6261.200346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6266.200328] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6273.397767] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6279.002698] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6285.419849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6285.718478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6285.724154] Lustre: Skipped 4 previous similar messages [ 6291.815571] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6299.165566] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6304.196892] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6304.526540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6309.034530] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6309.305536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6312.425298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6312.434246] LustreError: Skipped 1 previous similar message [ 6312.437922] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6312.459738] Lustre: Skipped 2 previous similar messages [ 6312.468580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6312.487939] Lustre: Skipped 7 previous similar messages [ 6317.432959] Lustre: server umount lustre-OST0000 complete [ 6323.678334] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6328.085954] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6357.086970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6364.423555] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6365.606799] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.621065] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6371.682181] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6374.400044] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6378.346428] Lustre: server umount lustre-OST0000 complete [ 6384.610900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6384.622924] LustreError: Skipped 1 previous similar message [ 6396.383139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6396.598741] Lustre: server umount lustre-MDT0000 complete [ 6396.895331] Lustre: 189180:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790745702/real 1790745702] req@00000000bd6470ce x1877732211546112/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790745718 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6396.938706] LustreError: 214229:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6396.970177] LustreError: 214229:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6401.274567] LustreError: 216498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790745723 with bad export cookie 13207279372057040550 [ 6401.282446] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6401.287280] LustreError: 216498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6402.123872] Lustre: server umount lustre-MDT0001 complete [ 6411.670438] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 01:22:11 (1790745731) [ 6418.823421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6424.222871] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6433.065512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6437.535870] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6445.823882] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6451.886394] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6459.505279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6465.728601] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6473.174899] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6478.094866] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6478.318370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6483.381933] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6483.687570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6491.024384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6491.210619] Lustre: lustre-OST0001: new disk, initializing [ 6491.212972] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6491.217766] Lustre: 221724:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6498.066773] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6499.490367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6499.512744] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6499.696862] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6509.648531] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6514.058775] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6514.237181] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6519.562693] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6519.734516] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6525.412074] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6525.416752] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6525.419237] LustreError: Skipped 1 previous similar message [ 6525.435870] Lustre: Skipped 4 previous similar messages [ 6525.438198] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6525.453272] Lustre: Skipped 10 previous similar messages [ 6527.520161] Lustre: server umount lustre-OST0001 complete [ 6533.997368] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6534.351982] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6539.284596] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6539.518132] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6567.638648] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6567.838205] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6567.849573] Lustre: Skipped 5 previous similar messages [ 6573.042710] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6573.053306] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6573.061946] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6573.086508] Lustre: Skipped 1 previous similar message [ 6574.513091] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6581.947868] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6583.397289] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6591.601395] Lustre: server umount lustre-OST0001 complete [ 6616.448528] Lustre: server umount lustre-OST0000 complete [ 6627.131903] Lustre: server umount lustre-MDT0000 complete [ 6630.908753] LustreError: 218336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790745952 with bad export cookie 13207279372057041502 [ 6630.909890] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6630.928926] LustreError: 218336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6645.216050] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6645.904851] Lustre: server umount lustre-MDT0001 complete [ 6673.435632] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 01:26:33 (1790745993) [ 6681.492475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6681.956947] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6681.977248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6687.294623] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6694.959433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6695.050629] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6695.089825] Lustre: 226967:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6699.667980] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6708.053519] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6714.929248] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6723.822850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6723.978301] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6731.900521] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6738.041621] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6746.859281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6746.902733] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6753.068572] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6760.388797] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6762.234534] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6767.047402] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6767.237268] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6771.979531] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6772.295421] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6778.630306] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ce1c6bf1000.ost_server_uuid 50 [ 6780.210719] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8ce1c6bf1000.ost_server_uuid in IDLE state after 0 sec [ 6781.171975] Lustre: Permanently deactivating lustre-OST0001 [ 6781.189307] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6785.089127] Lustre: Permanently reactivating lustre-OST0001 [ 6789.601178] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6789.612213] LustreError: Skipped 4 previous similar messages [ 6789.630431] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6789.665583] Lustre: Skipped 7 previous similar messages [ 6789.680543] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6789.691017] Lustre: Skipped 9 previous similar messages [ 6795.634996] Lustre: server umount lustre-OST0001 complete [ 6800.355605] LustreError: 231739:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6800.370566] LustreError: 231739:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 40 previous similar messages [ 6805.385755] Lustre: server umount lustre-OST0000 complete [ 6823.391741] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6823.624452] Lustre: server umount lustre-MDT0000 complete [ 6827.417364] LustreError: 226959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790746149 with bad export cookie 13207279372057042797 [ 6827.423242] LustreError: 226959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6827.424386] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6827.771544] Lustre: server umount lustre-MDT0001 complete [ 6849.491656] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 01:29:29 (1790746169) [ 6856.960404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6857.189918] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6857.220738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6857.227907] Lustre: Skipped 5 previous similar messages [ 6861.182141] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6869.879612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6869.981881] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6870.013247] Lustre: 233663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6876.246200] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6884.314665] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6890.158853] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6897.530028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6897.830242] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6897.887242] Lustre: Permanently deactivating lustre-OST0000 [ 6897.919683] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6903.747650] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6905.398746] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6912.372189] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6919.788631] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6919.857349] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6925.870439] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6926.009820] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6933.776030] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6935.601347] Lustre: Permanently reactivating lustre-OST0000 [ 6935.621247] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6935.627467] Lustre: Skipped 2 previous similar messages [ 6942.735851] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6942.739323] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 37 seconds. I think it's dead, and I am evicting it. exp 000000001007a3d8, cur 1790746264 deadline 1790746257 last 1790746227 [ 6942.740952] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6942.749060] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6942.855381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6952.723271] Lustre: server umount lustre-OST0001 complete [ 6971.359151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6971.685785] Lustre: server umount lustre-OST0000 complete [ 6983.100625] Lustre: server umount lustre-MDT0000 complete [ 6986.821313] LustreError: 233657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790746308 with bad export cookie 13207279372057044470 [ 6986.826646] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6986.828301] LustreError: 233657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6987.192921] Lustre: server umount lustre-MDT0001 complete [ 6997.784972] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 7000.857434] Key type lgssc unregistered [ 7001.180169] LNet: 238546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7001.184756] LNetError: 238546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7001.209384] LNet: Removed LNI 192.168.202.151@tcp [ 7002.004712] Key type .llcrypt unregistered [ 7002.007239] Key type ._llcrypt unregistered [ 7026.202559] Key type ._llcrypt registered [ 7026.204029] Key type .llcrypt registered [ 7026.307686] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 7027.561972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7027.580125] alg: No test for adler32 (adler32-zlib) [ 7028.610565] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 7028.893621] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 7030.655455] Key type lgssc registered [ 7031.996758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7042.290919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7042.308087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7043.958456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7048.275113] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7053.281951] LustreError: 240027:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7053.290818] LustreError: 240027:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7057.302625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7057.364346] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7057.401725] Lustre: 240016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7057.412103] Lustre: Permanently deactivating lustre-MDT0001 [ 7057.434429] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7057.717242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7063.133124] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7064.607762] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7071.178367] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7077.994883] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7084.217226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7084.855478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7088.960845] LustreError: 241873:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7088.979450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7092.053185] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7094.257464] LustreError: 242084:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7099.377413] LustreError: 242084:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7099.524133] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7103.491393] LustreError: 241871:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7103.530576] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7107.233311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7107.542285] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7112.696211] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7112.706149] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7113.676651] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7120.479375] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7123.326250] Lustre: Permanently reactivating lustre-MDT0001 [ 7123.360088] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7123.369225] Lustre: Skipped 2 previous similar messages [ 7129.059820] Lustre: lustre-MDT0001: Client 65fc4ad1-c5d5-4d03-bbef-430760517307 (at 192.168.202.51@tcp) reconnecting [ 7131.793993] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7131.810810] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7131.821406] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7131.839987] LustreError: 243647:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7131.841319] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7131.845847] LustreError: 243647:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7131.845967] LustreError: 243647:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.151@tcp: cfg command failed: rc = -108 [ 7131.846927] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7131.846927] [ 7131.888797] LustreError: 240020:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7136.075294] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7137.902392] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7144.973256] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7145.221271] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7152.334913] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7152.848327] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7155.557206] Lustre: Permanently deactivating lustre-MDT0001 [ 7155.579532] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7155.582754] Lustre: Skipped 2 previous similar messages [ 7163.090622] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7173.091299] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7173.111808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7173.114113] Lustre: Skipped 1 previous similar message [ 7173.148040] Lustre: Skipped 3 previous similar messages [ 7178.211339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7178.222092] Lustre: Skipped 3 previous similar messages [ 7179.119991] Lustre: server umount lustre-MDT0000 complete [ 7183.335769] LustreError: 240022:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7183.357302] LustreError: 240022:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7183.848270] LustreError: 240767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790746505 with bad export cookie 7827414194981608865 [ 7183.856585] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7183.866572] LustreError: 240767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7184.417208] Lustre: server umount lustre-MDT0001 complete [ 7190.445903] Lustre: server umount lustre-OST0000 complete [ 7201.581435] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 01:35:21 (1790746521) [ 7211.785993] Lustre: server umount lustre-OST0001 complete [ 7218.515971] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 7228.442584] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 7263.857622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7264.095591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7264.104937] Lustre: Skipped 2 previous similar messages [ 7264.124266] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7264.206706] Lustre: lustre-MDT0000: new disk, initializing [ 7264.286971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7264.306078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7268.854285] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7281.141809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7281.319731] Lustre: 248304:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7281.397907] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7281.402664] Lustre: Skipped 1 previous similar message [ 7281.486550] Lustre: lustre-MDT0001: new disk, initializing [ 7281.599631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7281.641872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7281.668824] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7286.698473] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7296.287416] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7302.776555] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7309.572522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7309.888866] Lustre: lustre-OST0000: new disk, initializing [ 7309.892147] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7309.896977] Lustre: 250272:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7309.954453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7316.547875] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7317.611698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7317.618842] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7317.727224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7326.545767] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7328.571937] LustreError: 248311:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7337.142431] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7337.462665] Lustre: lustre-OST0001: new disk, initializing [ 7337.468421] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7337.484533] Lustre: 251389:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7338.592310] LustreError: 248311:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7338.638022] LustreError: 250563:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7342.711639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7342.721611] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7342.897267] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7344.379612] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7348.663117] LustreError: 250563:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7353.246648] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7354.855754] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7358.431534] Lustre: lustre-OST0001: Not available for connect from 192.168.202.51@tcp (stopping) [ 7358.433153] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7358.435981] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7358.435991] Lustre: Skipped 2 previous similar messages [ 7358.440910] Lustre: Skipped 1 previous similar message [ 7358.490058] LustreError: Skipped 1 previous similar message [ 7363.536453] Lustre: lustre-OST0001: Not available for connect from 192.168.202.51@tcp (stopping) [ 7363.543953] Lustre: Skipped 1 previous similar message [ 7368.651232] Lustre: lustre-OST0001: Not available for connect from 192.168.202.51@tcp (stopping) [ 7368.678676] Lustre: Skipped 2 previous similar messages [ 7371.231167] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7371.427988] Lustre: server umount lustre-OST0001 complete [ 7373.756206] LustreError: 250258:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7389.674077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7389.677413] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7389.686199] LustreError: Skipped 1 previous similar message [ 7389.689165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7389.700558] Lustre: Skipped 2 previous similar messages [ 7389.707470] Lustre: Skipped 2 previous similar messages [ 7395.689490] Lustre: server umount lustre-OST0000 complete [ 7404.516782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7404.518231] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7404.539713] Lustre: Skipped 1 previous similar message [ 7409.633874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7409.638025] Lustre: Skipped 6 previous similar messages [ 7414.239391] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7414.614562] Lustre: server umount lustre-MDT0000 complete [ 7414.756758] LustreError: 252246:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7414.777780] LustreError: 252246:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 7418.066929] LustreError: 251387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790746739 with bad export cookie 7827414194981611679 [ 7418.071626] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7418.078591] LustreError: 251387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7424.491531] Lustre: server umount lustre-MDT0001 complete [ 7435.674142] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 7438.806845] Key type lgssc unregistered [ 7439.173508] LNet: 253428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7439.190828] LNetError: 253428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7440.229875] LNet: Removed LNI 192.168.202.151@tcp [ 7441.139277] Key type .llcrypt unregistered [ 7441.149050] Key type ._llcrypt unregistered [ 7482.273127] Key type ._llcrypt registered [ 7482.279253] Key type .llcrypt registered [ 7482.380771] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 7483.181403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7483.213351] alg: No test for adler32 (adler32-zlib) [ 7484.680889] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 7485.134126] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 7486.927923] Key type lgssc registered [ 7488.378715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7498.628665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7498.680865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7500.113773] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7500.141788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7500.277744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7506.628622] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7518.367137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7518.454535] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7518.468251] Lustre: 255648:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7518.687068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7523.799197] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7532.110455] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7537.518691] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7545.681281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7545.962633] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7546.218260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7552.177946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7552.730499] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7560.704329] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7613.408372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7613.426903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7613.438053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7613.922453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7613.934869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7613.950567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7619.044828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7619.170609] Lustre: server umount lustre-OST0000 complete [ 7623.356911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7635.616679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7636.037886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7642.573735] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7644.257960] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7644.281561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7644.284560] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7650.193792] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7690.724226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7690.726977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7690.732189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7690.739674] Lustre: Skipped 1 previous similar message [ 7690.760761] Lustre: Skipped 2 previous similar messages [ 7695.333371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7695.345596] Lustre: Skipped 1 previous similar message [ 7696.751791] Lustre: server umount lustre-OST0000 complete [ 7700.450546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7705.577850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7705.584371] Lustre: Skipped 3 previous similar messages [ 7706.773469] Lustre: server umount lustre-MDT0000 complete [ 7710.646843] LustreError: 255641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790747032 with bad export cookie 11807106075415519552 [ 7710.665278] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7710.984584] Lustre: server umount lustre-MDT0001 complete [ 7722.145630] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 7724.950494] Key type lgssc unregistered [ 7725.233083] LNet: 261074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7725.238843] LNetError: 261074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7725.259568] LNet: Removed LNI 192.168.202.151@tcp [ 7726.296181] Key type .llcrypt unregistered [ 7726.298573] Key type ._llcrypt unregistered [ 7748.916694] Key type ._llcrypt registered [ 7748.921912] Key type .llcrypt registered [ 7749.014449] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 7750.209591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7750.283855] alg: No test for adler32 (adler32-zlib) [ 7751.364499] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 7751.579502] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 7753.263459] Key type lgssc registered [ 7754.325525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7761.238709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7761.255327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7762.670269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7767.622939] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7772.142594] LustreError: 262507:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7772.173756] LustreError: 262507:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7776.600607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7777.041611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7782.126926] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7790.813541] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7797.487833] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7805.502229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7805.793337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7807.890218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7812.622786] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7819.987764] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7846.883845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7846.886751] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7846.888601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7846.907028] Lustre: Skipped 1 previous similar message [ 7850.417379] Lustre: server umount lustre-OST0000 complete [ 7857.122551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7857.137814] Lustre: Skipped 1 previous similar message [ 7857.145908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7857.162020] Lustre: Skipped 2 previous similar messages [ 7862.249622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7862.256353] Lustre: Skipped 1 previous similar message [ 7862.438771] Lustre: server umount lustre-MDT0000 complete [ 7866.281515] LustreError: 262488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790747188 with bad export cookie 15475461427642154363 [ 7866.321800] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7866.660894] Lustre: server umount lustre-MDT0001 complete [ 7877.815800] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 7880.633594] Key type lgssc unregistered [ 7880.954761] LNet: 266814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7880.964072] LNetError: 266814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7880.984594] LNet: Removed LNI 192.168.202.151@tcp [ 7881.925494] Key type .llcrypt unregistered [ 7881.927669] Key type ._llcrypt unregistered [ 7901.181879] Key type ._llcrypt registered [ 7901.185732] Key type .llcrypt registered [ 7901.304842] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 7902.512256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7902.522129] alg: No test for adler32 (adler32-zlib) [ 7903.532729] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 7903.730227] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 7905.487406] Key type lgssc registered [ 7906.927281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7914.602769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7914.637526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7916.327852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7921.598444] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7926.753313] LustreError: 268050:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7926.775916] LustreError: 268050:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7931.443921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7931.764756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7938.005550] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7947.109087] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7955.152419] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7963.254749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7963.733472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7970.548912] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7980.037706] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7987.334068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8003.044062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8003.046266] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8003.080293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8003.742815] Lustre: server umount lustre-OST0000 complete [ 8008.672342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8008.695819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8008.746573] Lustre: Skipped 1 previous similar message [ 8008.754367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8008.760831] Lustre: Skipped 1 previous similar message [ 8013.282506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8013.286860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8014.712380] Lustre: server umount lustre-MDT0000 complete [ 8018.405362] LustreError: 268780:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8018.427308] LustreError: 268780:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8019.498063] LustreError: 268032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790747341 with bad export cookie 1806553097036435510 [ 8019.514598] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8019.523902] LustreError: 268032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8020.040981] Lustre: server umount lustre-MDT0001 complete [ 8040.164218] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 8045.882431] Key type lgssc unregistered [ 8046.614667] LNet: 271764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8046.624374] LNetError: 271764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8046.656491] LNet: Removed LNI 192.168.202.151@tcp [ 8048.229914] Key type .llcrypt unregistered [ 8048.231745] Key type ._llcrypt unregistered [ 8078.544472] Key type ._llcrypt registered [ 8078.546323] Key type .llcrypt registered [ 8078.643651] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8079.703679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8079.742810] alg: No test for adler32 (adler32-zlib) [ 8080.932285] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 8081.219924] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 8082.879189] Key type lgssc registered [ 8084.394691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8095.293982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8095.336086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8097.002178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8103.023172] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8106.469767] LustreError: 273213:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8106.476921] LustreError: 273213:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8111.588557] LustreError: 273214:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8113.516869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8114.325434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8120.211301] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8129.496164] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8136.470764] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8143.233657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8143.661200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8151.264310] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8159.086993] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8166.213154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8196.068193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8196.095596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8196.115334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8197.098569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8197.105524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8197.109969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8198.565192] Lustre: server umount lustre-OST0000 complete [ 8206.306808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8206.311122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8206.318507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8206.321888] Lustre: Skipped 1 previous similar message [ 8209.426884] Lustre: server umount lustre-MDT0000 complete [ 8212.465538] LustreError: 273213:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8212.486399] LustreError: 273213:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8213.532540] LustreError: 273194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790747535 with bad export cookie 13484809033688127237 [ 8213.538528] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8213.548077] LustreError: 273194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8214.043843] Lustre: server umount lustre-MDT0001 complete [ 8228.832166] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 8232.572640] Key type lgssc unregistered [ 8232.979033] LNet: 277551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8232.991568] LNetError: 277551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8233.008148] LNet: Removed LNI 192.168.202.151@tcp [ 8233.989592] Key type .llcrypt unregistered [ 8233.992110] Key type ._llcrypt unregistered [ 8252.072332] Key type ._llcrypt registered [ 8252.076329] Key type .llcrypt registered [ 8252.153890] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8252.940849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8253.015000] alg: No test for adler32 (adler32-zlib) [ 8254.455987] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 8255.009884] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 8256.775246] Key type lgssc registered [ 8258.173858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8265.692338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8265.718523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8267.381240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8271.746564] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8277.991192] LustreError: 278801:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8278.030737] LustreError: 278801:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8281.225280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8281.610872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8286.494961] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8294.237207] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8300.841404] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8310.207061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8310.577113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8312.653733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8317.465361] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8326.039493] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8343.522573] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8343.528440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8343.540031] LustreError: Skipped 1 previous similar message [ 8343.543831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8343.569117] Lustre: Skipped 1 previous similar message [ 8343.607195] Lustre: Skipped 1 previous similar message [ 8348.607714] Lustre: server umount lustre-OST0000 complete [ 8353.775787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8353.802667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8358.376909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8358.381453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8358.394608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8359.772544] Lustre: server umount lustre-MDT0000 complete [ 8363.711858] LustreError: 280669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790747685 with bad export cookie 9446835197619005945 [ 8363.722679] LustreError: 280669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8363.729645] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8364.073606] Lustre: server umount lustre-MDT0001 complete [ 8376.226768] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 8379.167842] Key type lgssc unregistered [ 8379.565761] LNet: 282501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8379.574117] LNetError: 282501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8379.591460] LNet: Removed LNI 192.168.202.151@tcp [ 8380.596124] Key type .llcrypt unregistered [ 8380.600955] Key type ._llcrypt unregistered [ 8393.474022] Key type ._llcrypt registered [ 8393.476377] Key type .llcrypt registered [ 8393.680270] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8412.754808] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 8427.437912] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8428.450157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8428.487625] alg: No test for adler32 (adler32-zlib) [ 8429.823605] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 8430.083029] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 8431.824461] Key type lgssc registered [ 8433.211219] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8468.178473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8468.211699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8469.442296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8469.465508] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8469.534667] Lustre: lustre-MDT0000: new disk, initializing [ 8469.596932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8469.612885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8474.042537] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8484.463314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8484.596067] Lustre: 286076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8484.696969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8484.705654] Lustre: Skipped 1 previous similar message [ 8484.817970] Lustre: lustre-MDT0001: new disk, initializing [ 8484.967067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8485.052525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8485.070583] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8490.386860] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8502.245693] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8508.721885] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8517.498913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8517.857530] Lustre: lustre-OST0000: new disk, initializing [ 8517.863961] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8517.875616] Lustre: 288044:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8518.050945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8524.880637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8524.903506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8525.089303] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8526.421641] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8537.665380] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8542.604342] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8542.890633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8548.034157] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8548.306651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8556.002537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8556.010663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8556.026374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8556.046705] Lustre: Skipped 1 previous similar message [ 8561.121066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8561.125215] Lustre: Skipped 1 previous similar message [ 8565.215195] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8565.471549] Lustre: server umount lustre-OST0000 complete [ 8571.367611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8571.379733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8571.387913] Lustre: Skipped 1 previous similar message [ 8571.401285] Lustre: Skipped 1 previous similar message [ 8575.267235] Lustre: server umount lustre-MDT0000 complete [ 8576.500561] LustreError: 286086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8576.538702] LustreError: 286086:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8579.081672] LustreError: 288043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790747900 with bad export cookie 8901563259487588417 [ 8579.086200] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8579.103980] LustreError: 288043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8579.539426] Lustre: server umount lustre-MDT0001 complete [ 8587.788757] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 01:58:27 (1790747907) [ 8594.936691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8595.503126] LustreError: 290457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8595.561412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8601.038389] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8601.072597] LustreError: 290458:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8605.153164] LustreError: 290457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8610.084437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8610.334147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8615.004977] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8623.112391] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8629.525515] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8636.658784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8637.041880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8644.130855] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8654.332107] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8669.970218] Lustre: server umount lustre-OST0000 complete [ 8676.338129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8676.358054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8676.363159] Lustre: Skipped 2 previous similar messages [ 8681.441747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8681.456299] Lustre: Skipped 1 previous similar message [ 8684.613186] Lustre: server umount lustre-MDT0000 complete [ 8686.560468] LustreError: 290458:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8686.586355] LustreError: 290458:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8687.521318] Lustre: 284291:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790747993/real 1790747993] req@00000000ccad03c0 x1877735113832704/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790748009 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8687.576842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8687.596735] Lustre: Skipped 1 previous similar message [ 8688.323131] LustreError: 290437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790748010 with bad export cookie 8901563259487589299 [ 8688.347090] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8688.348169] LustreError: 290437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8688.688902] Lustre: server umount lustre-MDT0001 complete [ 8701.129192] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 8703.967962] Key type lgssc unregistered [ 8704.310733] LNet: 294013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8704.327466] LNetError: 294013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8704.343291] LNet: Removed LNI 192.168.202.151@tcp [ 8705.357285] Key type .llcrypt unregistered [ 8705.360634] Key type ._llcrypt unregistered [ 8733.646577] Key type ._llcrypt registered [ 8733.649550] Key type .llcrypt registered [ 8733.765486] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 8750.586949] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8751.999261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8752.105165] alg: No test for adler32 (adler32-zlib) [ 8753.465986] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 8753.775941] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 8755.511256] Key type lgssc registered [ 8757.313449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8799.715129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8799.730869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8800.973590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8800.998314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8801.057392] Lustre: lustre-MDT0000: new disk, initializing [ 8801.102904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8801.111366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8805.490972] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8815.844151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8815.978812] Lustre: 297868:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8816.027293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8816.031836] Lustre: Skipped 1 previous similar message [ 8816.091356] Lustre: lustre-MDT0001: new disk, initializing [ 8816.186656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8816.227717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8816.245699] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8821.220202] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8830.586807] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8837.179512] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8843.440095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8843.775595] Lustre: lustre-OST2710: new disk, initializing [ 8843.780569] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8843.790426] Lustre: 299837:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8843.945034] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8851.515372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8851.533855] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8852.096590] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8857.077334] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8862.066843] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8870.088200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8870.369117] Lustre: lustre-OST03e8: new disk, initializing [ 8870.373335] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8870.378753] Lustre: 300901:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8870.471910] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8878.010633] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8879.264534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8879.278922] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8879.407169] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8886.180904] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8894.152136] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8894.478535] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8900.616980] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8900.972936] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8905.540475] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8905.773235] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8911.273563] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8911.544736] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8918.501197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8918.509915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8918.538303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8920.550156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8920.562932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8920.579191] Lustre: Skipped 2 previous similar messages [ 8923.690517] Lustre: server umount lustre-MDT0000 complete [ 8927.476311] LustreError: 298671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790748249 with bad export cookie 17759039399802784667 [ 8927.486303] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8927.495975] LustreError: 298671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8927.850676] Lustre: server umount lustre-MDT0001 complete [ 8941.964281] Lustre: server umount lustre-OST2710 complete [ 8956.640840] Lustre: server umount lustre-OST03e8 complete [ 8981.201237] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 8992.594233] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9036.602685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9036.966449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9037.012988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9037.180737] Lustre: lustre-MDT0000: new disk, initializing [ 9037.241312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9037.255991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9042.042149] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9052.240105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9052.409118] Lustre: 306155:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9052.486540] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9052.494860] Lustre: Skipped 1 previous similar message [ 9052.618990] Lustre: lustre-MDT0001: new disk, initializing [ 9052.708628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9052.760485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9052.775870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9057.254315] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9067.007974] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9073.714624] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9082.357631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9082.674152] Lustre: lustre-OST0000: new disk, initializing [ 9082.683458] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9082.691696] Lustre: 308121:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9082.826229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9089.602037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9089.625766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9089.760359] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9090.896308] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9100.152855] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9104.676720] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9104.865959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9109.592889] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9109.794912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9115.627459] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9115.631929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9115.657072] Lustre: Skipped 2 previous similar messages [ 9115.670639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9115.673580] Lustre: Skipped 1 previous similar message [ 9117.747690] Lustre: server umount lustre-OST0000 complete [ 9124.321824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9124.326941] LustreError: Skipped 1 previous similar message [ 9124.332735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9124.347165] Lustre: Skipped 1 previous similar message [ 9124.352375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9124.360877] Lustre: Skipped 1 previous similar message [ 9127.507246] Lustre: server umount lustre-MDT0000 complete [ 9130.980653] LustreError: 306167:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9130.991625] LustreError: 306167:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 9131.327578] LustreError: 306148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790748453 with bad export cookie 17759039399802786361 [ 9131.344565] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9131.941564] Lustre: server umount lustre-MDT0001 complete [ 9140.466272] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 02:07:40 (1790748460) [ 9142.167513] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9144.105901] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 02:07:43 (1790748463) [ 9148.390703] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9174.909273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9175.095341] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9175.139832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9175.266900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9179.363318] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9187.949556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9188.035912] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9188.054577] Lustre: 311982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9192.169711] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9212.970401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9213.334550] Lustre: Denying initial registration attempt from nid 192.168.202.151@tcp, specified as failover [ 9213.354086] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9213.361647] LustreError: 313435:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9213.366447] LustreError: 313435:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9213.371755] LustreError: 313435:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9213.471602] Lustre: server umount lustre-OST0000 complete [ 9213.474668] LustreError: 313435:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9224.160715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9224.178205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9224.196781] Lustre: Skipped 1 previous similar message [ 9224.211202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9224.215810] Lustre: Skipped 2 previous similar messages [ 9227.393316] Lustre: server umount lustre-MDT0000 complete [ 9229.792534] LustreError: 311993:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9229.821511] LustreError: 311993:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9232.026472] LustreError: 311974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790748553 with bad export cookie 17759039399802787243 [ 9232.039549] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9232.041381] LustreError: 311974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9232.629926] Lustre: server umount lustre-MDT0001 complete [ 9246.988580] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 9251.385795] Key type lgssc unregistered [ 9251.772807] LNet: 314556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9251.784403] LNetError: 314556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9252.838337] LNet: Removed LNI 192.168.202.151@tcp [ 9253.948149] Key type .llcrypt unregistered [ 9253.953611] Key type ._llcrypt unregistered [ 9268.445735] Key type ._llcrypt registered [ 9268.447490] Key type .llcrypt registered [ 9268.540591] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9269.916638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9269.998165] alg: No test for adler32 (adler32-zlib) [ 9271.116348] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 9271.342626] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 9273.041047] Key type lgssc registered [ 9274.500899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9320.415679] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9328.276321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9328.303439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9329.484507] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9329.505390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9329.605104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9333.916543] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9341.929603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9342.060664] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9342.106822] Lustre: 317337:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9342.363957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9346.255928] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9353.322775] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9358.156213] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9377.996064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9378.304875] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9378.566606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9386.437954] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9395.913395] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9401.825462] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9401.833951] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9401.839244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9401.855236] Lustre: Skipped 1 previous similar message [ 9404.650535] Lustre: server umount lustre-OST0000 complete [ 9408.998217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9409.005799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9409.025853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9409.031260] Lustre: Skipped 1 previous similar message [ 9412.069241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9412.083504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9415.116483] Lustre: server umount lustre-MDT0000 complete [ 9418.987993] LustreError: 317329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790748740 with bad export cookie 6131863700707604229 [ 9418.990047] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9418.999231] LustreError: 317329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9419.346715] Lustre: server umount lustre-MDT0001 complete [ 9431.158873] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 9433.488888] Key type lgssc unregistered [ 9433.737464] LNet: 320909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9433.741486] LNetError: 320909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9433.751805] LNet: Removed LNI 192.168.202.151@tcp [ 9434.671419] Key type .llcrypt unregistered [ 9434.673363] Key type ._llcrypt unregistered [ 9460.099987] Key type ._llcrypt registered [ 9460.102837] Key type .llcrypt registered [ 9460.196441] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9461.418931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9461.496567] alg: No test for adler32 (adler32-zlib) [ 9462.549354] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 9462.791916] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 9464.519282] Key type lgssc registered [ 9465.766543] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9473.576191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9473.593235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9475.052864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9479.208028] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9484.261327] LustreError: 322346:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9484.274857] LustreError: 322346:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9486.961893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9487.358826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9491.643967] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9499.830365] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9505.865978] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9517.230534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9517.637860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9525.732813] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9533.787146] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9543.137748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9543.143605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9543.158282] Lustre: Skipped 1 previous similar message [ 9543.187988] Lustre: Skipped 2 previous similar messages [ 9546.732433] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.51@tcp (stopping) [ 9548.271031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9548.278332] Lustre: Skipped 2 previous similar messages [ 9549.636566] Lustre: server umount lustre-MDT0000 complete [ 9551.808673] LustreError: 323780:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9553.388378] LustreError: 322341:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9553.426888] LustreError: 322341:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9553.758311] LustreError: 322328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790748875 with bad export cookie 15878091356828425539 [ 9553.769788] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9556.946242] LustreError: 322342:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9556.971939] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.51@tcp (stopping) [ 9558.498117] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9558.513664] Lustre: Skipped 1 previous similar message [ 9562.046672] LustreError: 322342:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9562.051472] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.51@tcp (stopping) [ 9562.066207] LustreError: 322342:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9562.069413] Lustre: Skipped 1 previous similar message [ 9568.223973] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9568.462924] Lustre: server umount lustre-MDT0001 complete [ 9575.315238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9582.316390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9598.436409] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xdc5a4ff69af57b5e [ 9598.448875] Lustre: MGC192.168.202.151@tcp: Connection restored to 0@lo (at 0@lo) [ 9598.691545] LustreError: 324179:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9598.712665] LustreError: 324179:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9598.767197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9598.847146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9603.016775] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9612.973209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9613.284704] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9613.287838] Lustre: Skipped 3 previous similar messages [ 9613.446642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9618.265096] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9618.598624] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9624.922920] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9626.473784] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9633.889732] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9635.830306] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9644.009065] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9644.013785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9644.044784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9650.019494] Lustre: server umount lustre-OST0000 complete [ 9659.365156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9659.379699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9659.392191] Lustre: Skipped 1 previous similar message [ 9662.666189] Lustre: server umount lustre-MDT0000 complete [ 9664.993979] LustreError: 325947:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9665.028311] LustreError: 325947:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9666.346682] LustreError: 322326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790748988 with bad export cookie 15878091356828433246 [ 9666.368575] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9666.371183] LustreError: 322326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9666.758488] Lustre: server umount lustre-MDT0001 complete [ 9680.011352] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 9683.314884] Key type lgssc unregistered [ 9683.797121] LNet: 328615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9683.815417] LNetError: 328615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9683.848277] LNet: Removed LNI 192.168.202.151@tcp [ 9685.103822] Key type .llcrypt unregistered [ 9685.107948] Key type ._llcrypt unregistered [ 9719.305965] Key type ._llcrypt registered [ 9719.314788] Key type .llcrypt registered [ 9719.500545] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9721.118127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9721.427323] alg: No test for adler32 (adler32-zlib) [ 9722.532775] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [ 9722.836372] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 9724.809159] Key type lgssc registered [ 9726.481238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9736.944757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9736.999572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9738.678704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9743.342599] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9748.961755] LustreError: 330047:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9748.983327] LustreError: 330047:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9750.002524] LustreError: 330047:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9752.616581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9753.017434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9758.209796] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9766.570838] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9772.522739] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9777.638534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9777.644978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9777.651387] Lustre: Skipped 1 previous similar message [ 9781.960882] Lustre: server umount lustre-MDT0000 complete [ 9782.753127] LustreError: 330042:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9782.775225] LustreError: 330042:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9785.919801] LustreError: 330027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790749107 with bad export cookie 8673172973113621451 [ 9785.932029] LustreError: 330027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9785.934358] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9786.316488] Lustre: server umount lustre-MDT0001 complete [ 9804.123428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9804.488821] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9804.574809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9804.799266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9811.128135] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9822.883365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9823.010165] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9823.057804] Lustre: 332636:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9823.281276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9827.831404] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9835.539707] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9841.276317] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9851.324511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9851.666070] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9851.683275] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9851.704566] LustreError: 334522:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 9851.712591] LustreError: 334522:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9851.718388] LustreError: 334522:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9851.774882] Lustre: server umount lustre-OST0000 complete [ 9851.778089] LustreError: 334522:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9862.698168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9862.867603] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9863.072936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9869.213402] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9870.955694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9875.846936] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9883.307646] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9883.428949] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9883.559568] Lustre: lustre-OST0001: new disk, initializing [ 9883.565989] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9883.576864] Lustre: 336061:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9883.709131] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9888.856897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9888.863251] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9888.957516] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9889.454497] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9896.420277] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9904.610679] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9904.611753] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9904.616185] LustreError: Skipped 1 previous similar message [ 9904.627034] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9904.629359] Lustre: Skipped 1 previous similar message [ 9904.638108] Lustre: Skipped 2 previous similar messages [ 9905.740816] Lustre: server umount lustre-OST0001 complete [ 9909.734945] LustreError: 334989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9909.736327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9909.747114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9909.771079] LustreError: 334989:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9909.788038] Lustre: Skipped 1 previous similar message [ 9914.860613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9914.862915] LustreError: 334988:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9914.865995] Lustre: Skipped 2 previous similar messages [ 9914.895439] LustreError: 334988:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9919.970628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9919.985723] Lustre: Skipped 1 previous similar message [ 9923.039517] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9923.319916] Lustre: server umount lustre-OST0000 complete [ 9930.212809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9930.213524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9930.237973] Lustre: Skipped 1 previous similar message [ 9930.252913] Lustre: Skipped 1 previous similar message [ 9934.883682] Lustre: server umount lustre-MDT0000 complete [ 9935.335557] LustreError: 332644:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9935.368976] LustreError: 332644:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9938.797370] LustreError: 332629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790749260 with bad export cookie 8673172973113622004 [ 9938.808105] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9938.814019] LustreError: 332629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9939.205861] Lustre: server umount lustre-MDT0001 complete [ 9950.450877] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 9953.011476] Key type lgssc unregistered [ 9953.312889] LNet: 338020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9953.322439] LNetError: 338020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9954.342228] LNet: Removed LNI 192.168.202.151@tcp [ 9955.283128] Key type .llcrypt unregistered [ 9955.286099] Key type ._llcrypt unregistered [10025.156515] Key type ._llcrypt registered [10025.160924] Key type .llcrypt registered [10025.317048] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [10041.225603] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [10042.903595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10042.931646] alg: No test for adler32 (adler32-zlib) [10044.131701] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [10044.495379] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [10046.243339] Key type lgssc registered [10048.294871] Lustre: Echo OBD driver; http://www.lustre.org/ [10087.177776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10087.261754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10088.549989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10088.594936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10088.685252] Lustre: lustre-MDT0000: new disk, initializing [10088.750545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10088.772820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10093.572950] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10105.001725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10105.201119] Lustre: 342972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10105.288575] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10105.293705] Lustre: Skipped 1 previous similar message [10105.365722] Lustre: lustre-MDT0001: new disk, initializing [10105.473577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10105.507566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10105.513027] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10109.947563] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10120.825398] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10128.067457] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10135.510382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10135.779297] Lustre: lustre-OST0000: new disk, initializing [10135.784474] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10135.793759] Lustre: 344941:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10135.890187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10142.543588] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10144.436796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10144.439917] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10144.549285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10152.959463] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10158.525845] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10158.806489] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10164.267596] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10164.549527] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10170.354021] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10170.363290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10170.370474] LustreError: Skipped 1 previous similar message [10170.381192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10170.390084] Lustre: Skipped 1 previous similar message [10170.421973] Lustre: Skipped 1 previous similar message [10175.468488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10175.481932] Lustre: Skipped 1 previous similar message [10180.580335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10180.593411] Lustre: Skipped 1 previous similar message [10181.087580] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10181.319115] Lustre: server umount lustre-OST0000 complete [10187.242513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10187.272638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10187.302095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10190.817523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10192.392900] Lustre: server umount lustre-MDT0000 complete [10196.256725] LustreError: 342963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790749518 with bad export cookie 16773523434752852107 [10196.275898] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10196.795990] Lustre: server umount lustre-MDT0001 complete [10205.068823] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 02:25:25 (1790749525) [10212.569628] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 02:25:32 (1790749532) [10221.109677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10221.708283] LustreError: 347699:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10221.739787] LustreError: 347699:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10221.792308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10225.941720] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10227.172668] LustreError: 347700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10231.270368] LustreError: 347699:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10235.505026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10235.961601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10240.912602] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10248.704380] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10254.315937] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10261.399483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10261.497758] LustreError: 349522:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10261.742919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10268.203820] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10276.327281] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10283.866286] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10285.605425] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10287.202105] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10289.112457] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10290.958140] Lustre: Failing over lustre-MDT0000 [10291.263534] Lustre: server umount lustre-MDT0000 complete [10292.195550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10292.204770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10292.217889] LustreError: 347700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10302.944877] LustreError: 347699:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10302.976619] LustreError: 347699:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [10309.088240] Lustre: 341182:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790749614/real 1790749614] req@000000006189064b x1877736807274112/t0(0) o400->MGC192.168.202.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790749630 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10309.118641] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10310.211239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10319.332649] LustreError: 347696:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10319.373378] LustreError: 347696:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [10319.615421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10319.666450] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10320.323405] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10324.033633] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10324.976525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10325.044196] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10325.082667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10333.678544] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10335.944584] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10339.800398] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10342.869754] Lustre: Failing over lustre-MDT0000 [10343.085381] Lustre: server umount lustre-MDT0000 complete [10345.451944] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10345.460867] Lustre: Skipped 4 previous similar messages [10345.470494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10354.116414] LustreError: 347694:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10354.127042] LustreError: 347694:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [10361.826346] Lustre: 341183:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790749667/real 1790749667] req@00000000483546be x1877736807295232/t0(0) o400->MGC192.168.202.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790749683 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10361.856447] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10362.980267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10372.071075] LustreError: 341180:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fbc28e7c x1877736807301376/t0(0) o250->MGC192.168.202.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10372.504367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10372.573761] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10372.581127] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10377.496483] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10377.728046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10377.735947] Lustre: Skipped 2 previous similar messages [10377.754254] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10377.795914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10386.474813] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10387.977671] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10398.179141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10398.197648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10398.230538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10398.236528] Lustre: Skipped 4 previous similar messages [10402.118527] Lustre: server umount lustre-MDT0000 complete [10405.465139] LustreError: 347680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790749727 with bad export cookie 16773523434752854816 [10405.470808] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10405.476808] LustreError: 347680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10405.782227] Lustre: server umount lustre-MDT0001 complete [10420.077683] Lustre: server umount lustre-OST0000 complete [10435.295557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10435.682261] LustreError: 354042:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10435.704367] LustreError: 354042:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 21 previous similar messages [10440.663159] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10450.602813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10451.048444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10451.052945] Lustre: Skipped 1 previous similar message [10456.885942] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10466.201157] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10472.819763] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10479.271036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10480.643742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10485.662895] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10493.422385] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10506.224268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10506.229215] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10506.234181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10506.238633] Lustre: Skipped 3 previous similar messages [10509.112463] Lustre: server umount lustre-OST0000 complete [10519.890570] Lustre: server umount lustre-MDT0000 complete [10524.929614] LustreError: 354022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790749846 with bad export cookie 16773523434752855565 [10524.936874] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10524.949702] LustreError: 354022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10525.352640] Lustre: server umount lustre-MDT0001 complete [10539.576391] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [10542.331773] Key type lgssc unregistered [10542.613657] LNet: 357576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10542.619238] LNetError: 357576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10542.632555] LNet: Removed LNI 192.168.202.151@tcp [10543.737427] Key type .llcrypt unregistered [10543.744454] Key type ._llcrypt unregistered [10570.613721] Key type ._llcrypt registered [10570.619746] Key type .llcrypt registered [10570.817568] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [10572.457377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10572.501908] alg: No test for adler32 (adler32-zlib) [10573.574586] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [10573.791927] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [10575.423352] Key type lgssc registered [10577.006294] Lustre: Echo OBD driver; http://www.lustre.org/ [10585.147221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10585.164719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10586.672725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10590.412828] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10596.833822] LustreError: 359106:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10596.859454] LustreError: 359106:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10598.102098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10598.544399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10603.324931] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10610.074316] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10614.792258] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10620.911288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10621.383427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10627.210733] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10631.678488] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10635.050846] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10647.009647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10647.014193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10647.034546] Lustre: Skipped 1 previous similar message [10647.601439] Lustre: server umount lustre-MDT0000 complete [10651.540019] LustreError: 359087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790749973 with bad export cookie 2154304802764509443 [10651.550174] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10651.560596] LustreError: 359087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10651.998303] Lustre: server umount lustre-MDT0001 complete [10665.273766] Lustre: server umount lustre-OST0000 complete [10678.600979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10679.116822] LustreError: 362816:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10679.222538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10684.351590] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10684.404120] LustreError: 362817:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10689.514789] LustreError: 362816:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10694.315993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10694.644100] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10694.654473] Lustre: Skipped 2 previous similar messages [10694.838938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10700.356543] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10708.542375] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10713.947819] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10720.871148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10721.240423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10724.350130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10727.364478] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10734.605658] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10745.832470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10745.840451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10745.847355] Lustre: Skipped 1 previous similar message [10745.851741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10749.940912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10749.948318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10749.968400] Lustre: Skipped 1 previous similar message [10749.984510] Lustre: Skipped 2 previous similar messages [10752.017922] LustreError: 365496:0:(ldlm_resource.c:1227:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (000000005dae69e0) refcount nonzero (2) after lock cleanup; forcing cleanup. [10752.283463] Lustre: server umount lustre-MDT0000 complete [10755.050387] LustreError: 362816:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10755.072935] LustreError: 362816:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10756.035347] LustreError: 362798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790750077 with bad export cookie 2154304802764510556 [10756.049424] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10756.063246] LustreError: 362798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10756.533105] Lustre: server umount lustre-MDT0001 complete [10770.556828] Lustre: server umount lustre-OST0000 complete [10784.437942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10784.829516] LustreError: 366729:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10784.923346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10788.550922] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10796.700229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10801.767196] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10809.871485] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10816.324898] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10825.774212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10826.279107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10826.285883] Lustre: Skipped 1 previous similar message [10829.393673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10835.230809] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10845.436486] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10851.818745] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10851.824110] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10851.857163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10857.432520] Lustre: server umount lustre-OST0000 complete [10862.053743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10862.063727] Lustre: Skipped 1 previous similar message [10862.084157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10862.099260] Lustre: Skipped 4 previous similar messages [10875.359155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10875.673392] Lustre: server umount lustre-MDT0000 complete [10877.416935] LustreError: 366730:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10877.440567] LustreError: 366730:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10879.605643] LustreError: 366709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790750201 with bad export cookie 2154304802764511578 [10879.623721] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10879.922373] Lustre: server umount lustre-MDT0001 complete [10892.071293] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [10894.799472] Key type lgssc unregistered [10895.136955] LNet: 370261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10895.153221] LNetError: 370261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10895.175697] LNet: Removed LNI 192.168.202.151@tcp [10896.233293] Key type .llcrypt unregistered [10896.235432] Key type ._llcrypt unregistered [10922.649070] Key type ._llcrypt registered [10922.652314] Key type .llcrypt registered [10922.755971] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [10924.143042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10924.241261] alg: No test for adler32 (adler32-zlib) [10925.336374] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [10925.763442] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [10927.528171] Key type lgssc registered [10929.068223] Lustre: Echo OBD driver; http://www.lustre.org/ [10936.258268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10936.293601] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10936.305065] LustreError: 371717:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10936.332427] LustreError: 371717:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10936.340742] LustreError: 371717:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10936.352488] LustreError: 371717:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10936.362805] LustreError: 371717:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10944.624903] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10944.639722] LustreError: 372114:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10944.659975] LustreError: 372114:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10944.664651] LustreError: 372114:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10944.669345] LustreError: 372114:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10944.674230] LustreError: 372114:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10964.399736] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [10966.901264] Key type lgssc unregistered [10967.160032] LNet: 373117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10967.163766] LNetError: 373117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10967.181755] LNet: Removed LNI 192.168.202.151@tcp [10968.283500] Key type .llcrypt unregistered [10968.285263] Key type ._llcrypt unregistered [10987.764315] Key type ._llcrypt registered [10987.771083] Key type .llcrypt registered [10987.896347] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [11003.361600] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [11004.712517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11004.786441] alg: No test for adler32 (adler32-zlib) [11005.811618] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [11006.030223] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [11007.695303] Key type lgssc registered [11009.024955] Lustre: Echo OBD driver; http://www.lustre.org/ [11048.490334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11048.536490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11049.846971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11049.896551] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11050.047421] Lustre: lustre-MDT0000: new disk, initializing [11050.205937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11050.238358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11055.100986] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11066.762599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11066.954484] Lustre: 376290:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11067.008252] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11067.011776] Lustre: Skipped 1 previous similar message [11067.075448] Lustre: lustre-MDT0001: new disk, initializing [11067.171838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11067.214463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11067.234754] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11072.293539] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11082.848816] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11088.828475] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11096.743129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11097.026612] Lustre: lustre-OST0000: new disk, initializing [11097.028960] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11097.033567] Lustre: 378264:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11097.089272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11103.332788] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11106.423220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11106.453490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11106.608948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11112.623785] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11117.836299] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11118.107949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11123.379310] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11123.685082] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11127.264709] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11127.278820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11127.282302] LustreError: Skipped 1 previous similar message [11127.284874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11127.331973] Lustre: Skipped 1 previous similar message [11132.391876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11132.406398] Lustre: Skipped 2 previous similar messages [11137.510458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11137.527598] Lustre: Skipped 1 previous similar message [11140.063159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11140.210460] Lustre: server umount lustre-OST0000 complete [11144.162593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11144.171534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11144.183114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11147.748437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11150.126416] Lustre: server umount lustre-MDT0000 complete [11153.974857] LustreError: 376280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790750475 with bad export cookie 18043537770775917281 [11153.984825] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11153.990643] LustreError: 376280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11154.293785] Lustre: server umount lustre-MDT0001 complete [11162.354581] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11164.813218] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 02:41:24 (1790750484) [11172.472576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11172.777240] LustreError: 380725:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11172.789829] LustreError: 380725:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11172.833817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11177.306380] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11177.965191] LustreError: 380726:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11183.072414] LustreError: 380725:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11186.140541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11186.553807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11191.763165] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11199.234985] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11204.723472] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11211.508493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11211.906177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11217.887428] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11225.555386] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11232.741884] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11232.887593] Lustre: lustre-OST0001: new disk, initializing [11232.891494] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11232.897343] Lustre: 383559:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11232.973769] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11237.835887] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11242.590924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11242.600145] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11242.700585] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11246.728908] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11253.231618] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11253.234818] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11253.249851] Lustre: Skipped 1 previous similar message [11253.249965] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11253.249971] Lustre: Skipped 2 previous similar messages [11253.286802] LustreError: Skipped 1 previous similar message [11256.570025] Lustre: server umount lustre-OST0001 complete [11258.342872] LustreError: 382548:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11263.470518] LustreError: 382546:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11263.486478] LustreError: 382546:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11273.713712] LustreError: 384483:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11273.738936] LustreError: 384483:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11274.720585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11274.730716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11274.747870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11274.750733] Lustre: Skipped 2 previous similar messages [11280.221658] Lustre: server umount lustre-OST0000 complete [11289.057625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11289.061514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11289.085037] Lustre: Skipped 1 previous similar message [11289.116304] LustreError: Skipped 1 previous similar message [11291.304580] Lustre: server umount lustre-MDT0000 complete [11294.182289] LustreError: 380726:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11294.206857] LustreError: 380726:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11295.609381] LustreError: 380707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790750617 with bad export cookie 18043537770775918163 [11295.619188] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11295.627736] LustreError: 380707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11302.207300] Lustre: server umount lustre-MDT0001 complete [11314.409527] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [11317.441297] Key type lgssc unregistered [11317.820166] LNet: 385549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11317.848566] LNetError: 385549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11317.878618] LNet: Removed LNI 192.168.202.151@tcp [11319.174456] Key type .llcrypt unregistered [11319.176466] Key type ._llcrypt unregistered [11382.035731] Key type ._llcrypt registered [11382.041867] Key type .llcrypt registered [11382.129632] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [11383.502334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11383.537491] alg: No test for adler32 (adler32-zlib) [11384.661593] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [11385.007134] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [11386.711380] Key type lgssc registered [11388.188862] Lustre: Echo OBD driver; http://www.lustre.org/ [11397.681096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11397.693963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11398.895510] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11398.912426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11398.999152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11403.257368] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11412.006735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11412.263858] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11412.336681] Lustre: 388368:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11412.591751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11418.669899] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11426.825023] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11432.648850] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11436.000463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11436.002605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11436.019976] Lustre: Skipped 1 previous similar message [11436.043492] Lustre: Skipped 1 previous similar message [11440.700058] Lustre: server umount lustre-MDT0000 complete [11444.621358] LustreError: 388360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790750766 with bad export cookie 9804029429580237543 [11444.630756] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11444.643554] LustreError: 388360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11444.964984] Lustre: server umount lustre-MDT0001 complete [11453.940667] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 02:46:13 (1790750773) [11459.906346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11460.419446] LustreError: 390882:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11460.438247] LustreError: 390882:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11460.487784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11464.438663] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11465.699953] LustreError: 390883:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11470.823500] LustreError: 390882:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11473.228515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11473.580247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11478.212716] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11485.581831] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11493.304825] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11502.544461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11502.836323] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11503.215915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11510.895326] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11521.116996] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11531.980709] Lustre: Permanently deactivating lustre-OST0000 [11532.015051] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11539.506582] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11541.826887] LustreError: 393869:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11541.838583] LustreError: 393869:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11541.846354] LustreError: 393869:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11546.084885] LustreError: 393919:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11546.097206] LustreError: 393919:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11554.938649] Lustre: server umount lustre-OST0000 complete [11558.255598] LustreError: 394169:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11558.262508] LustreError: 394169:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11560.931807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11560.940798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11560.967763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11562.992186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11562.997870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11563.041438] Lustre: Skipped 1 previous similar message [11568.103059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11570.293273] Lustre: server umount lustre-MDT0000 complete [11573.224991] LustreError: 391896:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11573.247561] LustreError: 391896:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11574.509793] LustreError: 390864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790750896 with bad export cookie 9804029429580238124 [11574.527124] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11574.528427] LustreError: 390864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11585.166852] Lustre: server umount lustre-MDT0001 complete [11593.395802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11597.431969] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11600.157755] LustreError: 395329:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11606.162380] Lustre: 395668:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.151@tcp [11608.043338] Lustre: server umount MGS complete [11619.732790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11620.307610] LustreError: 396392:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11620.343480] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11620.356414] Lustre: Skipped 1 previous similar message [11620.422322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11624.711155] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11630.562078] LustreError: 396393:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11630.598165] LustreError: 396393:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11634.575842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11634.933843] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11640.043312] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11648.153457] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11653.622222] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11655.492376] Lustre: Permanently reactivating lustre-OST0000 [11655.502787] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11655.507307] Lustre: Skipped 2 previous similar messages [11682.783364] LustreError: 398174:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11690.078136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11690.347693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11690.356766] Lustre: Skipped 1 previous similar message [11696.055405] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11702.348623] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11707.879874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11707.893031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11707.894335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11707.899539] Lustre: Skipped 2 previous similar messages [11712.600702] Lustre: server umount lustre-OST0000 complete [11717.089106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11717.097515] LustreError: Skipped 1 previous similar message [11717.105258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11717.119491] Lustre: Skipped 1 previous similar message [11717.129237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11722.723833] Lustre: server umount lustre-MDT0000 complete [11723.232900] LustreError: 396393:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11723.248492] LustreError: 396393:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11726.393107] LustreError: 400005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790751048 with bad export cookie 9804029429580239174 [11726.408754] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11726.412145] LustreError: 400005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11726.994422] Lustre: server umount lustre-MDT0001 complete [11738.817565] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [11741.494968] Key type lgssc unregistered [11741.916277] LNet: 400756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11741.924264] LNetError: 400756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11741.937714] LNet: Removed LNI 192.168.202.151@tcp [11742.735408] Key type .llcrypt unregistered [11742.741338] Key type ._llcrypt unregistered [11762.369310] Key type ._llcrypt registered [11762.370642] Key type .llcrypt registered [11762.458050] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [11778.613635] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [11780.280747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11780.322684] alg: No test for adler32 (adler32-zlib) [11781.359884] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [11781.577386] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [11783.247364] Key type lgssc registered [11784.834576] Lustre: Echo OBD driver; http://www.lustre.org/ [11819.525593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11819.587174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11821.058831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11821.124805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11821.335202] Lustre: lustre-MDT0000: new disk, initializing [11821.502456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11821.542105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11826.297383] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11835.861315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11835.954794] Lustre: 403927:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11836.001684] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11836.005190] Lustre: Skipped 1 previous similar message [11836.079318] Lustre: lustre-MDT0001: new disk, initializing [11836.193589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11836.231305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11836.240420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11839.893181] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11849.458937] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11855.064890] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11861.548663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11861.756132] Lustre: lustre-OST0000: new disk, initializing [11861.759253] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11861.763962] Lustre: 405895:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11861.841419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11867.489505] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11869.222403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11869.228161] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11869.371729] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11876.770656] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11881.650066] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11881.856871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11886.653872] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11886.834279] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11889.648387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11889.652777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11889.672040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11890.157819] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11890.166863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11890.178286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11890.183077] Lustre: Skipped 1 previous similar message [11894.845499] Lustre: server umount lustre-OST0000 complete [11900.387988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11900.394276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11900.401739] Lustre: Skipped 1 previous similar message [11900.426676] Lustre: Skipped 1 previous similar message [11904.414766] Lustre: server umount lustre-MDT0000 complete [11908.885745] LustreError: 403919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790751230 with bad export cookie 4567731145605960342 [11908.891611] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11908.897489] LustreError: 403919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11909.253652] Lustre: server umount lustre-MDT0001 complete [11918.550727] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11920.248558] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 02:54:00 (1790751240) [11926.571259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11927.078457] LustreError: 408356:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11927.100415] LustreError: 408356:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11927.160722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11932.058810] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11932.645344] LustreError: 408357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11937.770838] LustreError: 408356:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11940.210044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11940.730522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11945.131804] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11953.230761] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11958.096555] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11964.867208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11965.127807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11971.062795] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11978.142806] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11979.766030] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11982.180268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11986.913183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11986.925088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11986.963605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11988.984089] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11993.121639] Lustre: server umount lustre-OST0000 complete [11997.152759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11997.164170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11997.183700] Lustre: Skipped 1 previous similar message [11997.188186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11997.196701] Lustre: Skipped 2 previous similar messages [12003.474661] Lustre: server umount lustre-MDT0000 complete [12004.325385] LustreError: 408357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12004.350513] LustreError: 408357:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12007.851833] LustreError: 408337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790751329 with bad export cookie 4567731145605961224 [12007.859695] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12007.871614] LustreError: 408337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12008.393310] Lustre: server umount lustre-MDT0001 complete [12023.310461] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12027.087323] Key type lgssc unregistered [12027.432251] LNet: 412082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12027.440245] LNetError: 412082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12027.461892] LNet: Removed LNI 192.168.202.151@tcp [12028.645298] Key type .llcrypt unregistered [12028.648507] Key type ._llcrypt unregistered [12056.386286] Key type ._llcrypt registered [12056.387487] Key type .llcrypt registered [12056.458527] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12056.694621] Key type .llcrypt unregistered [12056.695841] Key type ._llcrypt unregistered [12072.430732] Key type ._llcrypt registered [12072.435256] Key type .llcrypt registered [12072.515799] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [12073.367367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12073.396349] alg: No test for adler32 (adler32-zlib) [12074.531832] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [12074.822435] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [12076.511197] Key type lgssc registered [12077.918753] Lustre: Echo OBD driver; http://www.lustre.org/ [12086.061227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12086.100884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12087.587674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12092.315728] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12098.017240] LustreError: 414307:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12098.037078] LustreError: 414307:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12101.428872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12101.859614] LustreError: 414308:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12101.883176] LustreError: 414308:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12101.908431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12107.236839] LustreError: 414307:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12107.264621] LustreError: 414307:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12110.487642] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12112.357915] LustreError: 414303:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12112.369446] LustreError: 414303:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12112.372711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12117.481322] LustreError: 414308:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12117.505366] LustreError: 414308:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12117.947599] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12125.275719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12125.672494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12130.417959] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12140.365689] Lustre: server umount lustre-OST0000 complete [12146.147072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12146.153633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12146.165220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12146.185750] Lustre: Skipped 1 previous similar message [12151.271454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12151.291070] Lustre: Skipped 1 previous similar message [12153.467885] Lustre: server umount lustre-MDT0000 complete [12156.384040] Lustre: 413608:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790751462/real 1790751462] req@00000000113b6faa x1877738935838208/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790751478 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12156.387386] LustreError: 414308:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12156.407856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12156.421298] LustreError: 414308:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12156.447247] Lustre: Skipped 1 previous similar message [12156.938453] LustreError: 416701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790751478 with bad export cookie 1328147099369900729 [12156.945311] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12156.957124] LustreError: 416701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12157.292092] Lustre: server umount lustre-MDT0001 complete [12167.741435] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12170.569732] Key type lgssc unregistered [12170.807496] LNet: 417453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12170.814250] LNetError: 417453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12170.827358] LNet: Removed LNI 192.168.202.151@tcp [12171.888740] Key type .llcrypt unregistered [12171.893305] Key type ._llcrypt unregistered [12195.510803] Key type ._llcrypt registered [12195.513188] Key type .llcrypt registered [12195.593638] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [12196.598135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12196.626264] alg: No test for adler32 (adler32-zlib) [12197.637361] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [12197.866214] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [12199.535853] Key type lgssc registered [12200.752849] Lustre: Echo OBD driver; http://www.lustre.org/ [12207.711330] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12207.736721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12223.264165] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12228.385721] LustreError: 418863:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.151@tcp: failed processing log, type 4: rc = -110 [12255.071238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12261.184891] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12268.298298] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12276.407511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12277.210276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12277.338732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12282.343107] LustreError: 418889:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12282.396876] LustreError: 418889:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12283.232554] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12287.477181] LustreError: 418888:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12289.527216] LustreError: 419903:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12289.563344] LustreError: 419903:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12294.644525] LustreError: 418889:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12294.685396] LustreError: 418889:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12296.635957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12297.296629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12302.657491] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12310.335625] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12315.604592] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12322.785728] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12322.797130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12322.825124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12322.840159] Lustre: Skipped 1 previous similar message [12324.925298] Lustre: server umount lustre-OST0000 complete [12330.989628] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12330.999609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12331.031245] Lustre: Skipped 1 previous similar message [12331.045058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12333.035793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12333.039145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12335.147607] Lustre: server umount lustre-MDT0000 complete [12338.144984] LustreError: 419907:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12338.157968] LustreError: 419907:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12338.713349] LustreError: 418870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790751660 with bad export cookie 13243337976450416653 [12338.715872] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12338.723286] LustreError: 418870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12339.075454] Lustre: server umount lustre-MDT0001 complete [12351.001267] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12354.058301] Key type lgssc unregistered [12354.366865] LNet: 422445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12354.376927] LNetError: 422445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12354.411410] LNet: Removed LNI 192.168.202.151@tcp [12355.289410] Key type .llcrypt unregistered [12355.291054] Key type ._llcrypt unregistered [12380.654855] Key type ._llcrypt registered [12380.657748] Key type .llcrypt registered [12380.774263] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [12381.697057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12381.744703] alg: No test for adler32 (adler32-zlib) [12382.880114] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [12383.113122] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [12384.807226] Key type lgssc registered [12386.215828] Lustre: Echo OBD driver; http://www.lustre.org/ [12394.646782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12394.686189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12396.206280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12401.905674] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12406.756477] LustreError: 423877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12406.776727] LustreError: 423877:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12410.732698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12411.355087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12416.117839] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12425.262542] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12431.146494] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12438.723614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12439.027427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12442.111179] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12446.341814] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12453.788300] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12460.412792] Lustre: server umount lustre-MDT0000 complete [12461.016149] LustreError: 423874:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12461.537832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12461.557343] Lustre: Skipped 1 previous similar message [12463.243735] LustreError: 423857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.51@tcp arrived at 1790751785 with bad export cookie 16985918748599087018 [12465.236223] LustreError: 423857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790751787 with bad export cookie 16985918748599086129 [12465.255413] LustreError: 423857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12465.259119] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12466.661313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12466.667657] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12466.668563] LustreError: 423874:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12466.668570] LustreError: 423874:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12466.688686] Lustre: Skipped 1 previous similar message [12466.693170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12470.688846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12470.691427] LustreError: 423878:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12470.736221] LustreError: 423878:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12471.540597] Lustre: server umount lustre-OST0000 complete [12475.808942] LustreError: 423877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12475.848397] LustreError: 423877:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12479.650157] Lustre: server umount lustre-MDT0001 complete [12489.868388] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12492.242170] Key type lgssc unregistered [12492.597130] LNet: 427564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12492.602859] LNetError: 427564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12492.618147] LNet: Removed LNI 192.168.202.151@tcp [12493.475942] Key type .llcrypt unregistered [12493.482390] Key type ._llcrypt unregistered [12517.400660] Key type ._llcrypt registered [12517.402702] Key type .llcrypt registered [12517.506139] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [12519.003833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12519.062869] alg: No test for adler32 (adler32-zlib) [12520.224764] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [12520.460922] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [12522.167330] Key type lgssc registered [12523.433644] Lustre: Echo OBD driver; http://www.lustre.org/ [12531.154839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12531.171467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12532.667894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12537.320976] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12542.945482] LustreError: 428996:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12542.969522] LustreError: 428996:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12547.012435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12547.681297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12551.538672] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12559.367500] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12565.501437] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12572.762161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12573.206922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12577.339813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12579.374037] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12586.593739] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12592.616562] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12592.624968] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12592.631047] Lustre: Skipped 1 previous similar message [12592.637940] Lustre: Skipped 1 previous similar message [12594.643284] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.51@tcp (stopping) [12597.752030] LustreError: 429724:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12597.772887] LustreError: 429724:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12597.932774] Lustre: server umount lustre-MDT0001 complete [12599.743863] LustreError: 430808:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12602.859709] LustreError: 428995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12602.889609] LustreError: 428995:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12607.970100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12607.978302] LustreError: 428996:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12607.997420] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12608.026524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12611.566359] Lustre: server umount lustre-OST0000 complete [12615.810675] Lustre: server umount lustre-MDT0000 complete [12628.564652] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12631.536559] Key type lgssc unregistered [12631.820721] LNet: 432678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12631.839294] LNetError: 432678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12631.859113] LNet: Removed LNI 192.168.202.151@tcp [12632.752568] Key type .llcrypt unregistered [12632.758504] Key type ._llcrypt unregistered [12659.169412] Key type ._llcrypt registered [12659.171698] Key type .llcrypt registered [12659.350839] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [12677.623123] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [12679.254489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12679.316542] alg: No test for adler32 (adler32-zlib) [12680.453395] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [12680.767725] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [12682.511358] Key type lgssc registered [12683.889451] Lustre: Echo OBD driver; http://www.lustre.org/ [12719.617904] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [12734.647945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12734.694830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12736.001210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12736.040323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12736.131981] Lustre: lustre-MDT0000: new disk, initializing [12736.194192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12736.211429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12740.807852] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12758.252620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12758.613458] Lustre: lustre-OST0000: new disk, initializing [12758.628778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12758.635938] Lustre: Skipped 1 previous similar message [12758.669557] Lustre: 437730:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12758.792288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12767.962342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12767.974616] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12768.025059] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12768.356332] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12780.930699] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12797.414594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12797.543858] Lustre: 436711:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12797.604292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12797.723232] Lustre: lustre-MDT0001: new disk, initializing [12797.813328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12797.858862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12797.877225] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12801.893318] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12809.974515] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12810.138567] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12814.483813] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12814.778943] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12820.947720] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.51@tcp (stopping) [12822.498131] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12822.501059] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12822.514929] Lustre: Skipped 1 previous similar message [12822.528401] Lustre: Skipped 1 previous similar message [12825.894932] Lustre: server umount lustre-MDT0001 complete [12831.171518] LustreError: 436721:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12831.206583] LustreError: 436721:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12832.751318] LustreError: 436724:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12832.773355] LustreError: 436724:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12837.858683] LustreError: 439132:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12837.881712] LustreError: 439132:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12842.979329] LustreError: 436719:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12843.013328] LustreError: 436719:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12845.024275] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12845.035221] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12845.046063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12848.098276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12848.109508] LustreError: 436725:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12853.220779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12858.337824] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12858.339179] LustreError: 436724:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12858.384458] LustreError: 436724:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12858.638451] Lustre: server umount lustre-OST0000 complete [12869.076906] Lustre: server umount lustre-MDT0000 complete [12883.467582] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12886.578707] Key type lgssc unregistered [12886.976796] LNet: 441477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12886.984822] LNetError: 441477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12888.039516] LNet: Removed LNI 192.168.202.151@tcp [12889.190942] Key type .llcrypt unregistered [12889.194675] Key type ._llcrypt unregistered [12929.937960] Key type ._llcrypt registered [12929.941442] Key type .llcrypt registered [12930.051677] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12930.341528] Key type .llcrypt unregistered [12930.344100] Key type ._llcrypt unregistered [12952.484094] Key type ._llcrypt registered [12952.490508] Key type .llcrypt registered [12952.615552] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [12953.847129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12953.935580] alg: No test for adler32 (adler32-zlib) [12955.511213] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [12956.094290] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [12957.911686] Key type lgssc registered [12959.422960] Lustre: Echo OBD driver; http://www.lustre.org/ [12987.805778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12987.859158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12989.140304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12989.192847] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12989.281807] Lustre: lustre-MDT0000: new disk, initializing [12989.383166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12989.406964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12993.973963] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13006.145869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13006.298144] Lustre: 444867:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13006.349936] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13006.354916] Lustre: Skipped 1 previous similar message [13006.423615] Lustre: lustre-MDT0001: new disk, initializing [13006.585150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13006.645311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13006.660197] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13011.070556] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13022.741294] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13029.644935] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13036.318859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13036.627219] Lustre: lustre-OST0000: new disk, initializing [13036.632102] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13036.643514] Lustre: 446838:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13036.737301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13042.394976] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13043.291578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13043.314370] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13043.483890] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13051.348815] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13059.047837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13059.058885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13059.061013] Lustre: Skipped 1 previous similar message [13059.076514] Lustre: Skipped 2 previous similar messages [13064.160916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13064.174851] Lustre: Skipped 2 previous similar messages [13064.421194] Lustre: server umount lustre-MDT0000 complete [13067.815872] LustreError: 444860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790752389 with bad export cookie 9826321778111166532 [13067.822423] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13067.831814] LustreError: 444860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13068.211893] Lustre: server umount lustre-MDT0001 complete [13082.148595] Lustre: server umount lustre-OST0000 complete [13091.447381] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 03:13:31 (1790752411) [13098.835634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13099.470506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13103.978661] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13104.608709] LustreError: 448866:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13104.628070] LustreError: 448866:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13109.741908] LustreError: 448867:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13111.979824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13112.334392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13116.941737] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13124.377964] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13130.595037] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13151.105946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13151.499568] Lustre: Found index 0 for lustre-OST0000, updating log [13151.966331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13158.503744] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13165.713847] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13170.440403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13178.853754] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13178.862789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13178.871245] Lustre: Skipped 1 previous similar message [13178.875687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13182.626314] Lustre: server umount lustre-OST0000 complete [13199.337656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13199.348567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13199.364980] Lustre: Skipped 1 previous similar message [13199.367986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13199.372089] Lustre: Skipped 3 previous similar messages [13201.380081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13204.555728] Lustre: server umount lustre-MDT0000 complete [13205.473747] LustreError: 449868:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13205.503679] LustreError: 449868:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13208.195593] LustreError: 448846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790752530 with bad export cookie 9826321778111167918 [13208.204584] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13208.208664] LustreError: 448846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13208.639737] Lustre: server umount lustre-MDT0001 complete [13217.209132] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 03:15:37 (1790752537) [13220.320024] LNet: 1 local NIs in recovery (showing 1): 192.168.202.151@tcp [13842.930646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13843.292810] LustreError: 455277:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13843.304752] LustreError: 455277:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13843.342692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13848.563052] LustreError: 455278:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13849.104583] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13853.667177] LustreError: 455277:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13859.487291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13859.798869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13864.512211] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13873.104157] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13878.813228] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13898.207388] LNet: 1 local NIs in recovery (showing 1): 192.168.202.151@tcp [13899.327473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13899.599244] Lustre: Found index 0 for lustre-OST0000, updating log [13899.885301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13906.769725] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13914.037715] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13916.326609] Lustre: server umount lustre-OST0000 complete [13921.186575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13921.192484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13921.198151] Lustre: Skipped 1 previous similar message [13921.213314] Lustre: Skipped 3 previous similar messages [13922.213831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13922.223309] Lustre: Skipped 1 previous similar message [13923.233219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13923.242543] Lustre: Skipped 1 previous similar message [13925.290538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13925.305178] Lustre: Skipped 2 previous similar messages [13926.497852] Lustre: server umount lustre-MDT0000 complete [13927.468327] LustreError: 455272:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13927.499514] LustreError: 455272:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13931.561012] LustreError: 455277:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13931.610518] LustreError: 455277:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [13932.367312] LustreError: 455259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790753254 with bad export cookie 9826321778111169010 [13932.369988] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13932.376448] LustreError: 455259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13932.657882] Lustre: server umount lustre-MDT0001 complete [13958.625727] LNet: 1 local NIs in recovery (showing 1): 192.168.202.151@tcp [13963.873647] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [13974.619544] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [14011.237527] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 03:28:51 (1790753331) [14030.157804] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [14033.280571] Key type lgssc unregistered [14033.654674] LNet: 461887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14033.659657] LNetError: 461887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14161.703512] LNet: Removed LNI 192.168.202.151@tcp [14162.670612] Key type .llcrypt unregistered [14162.673799] Key type ._llcrypt unregistered [14177.940294] Key type ._llcrypt registered [14177.944417] Key type .llcrypt registered [14178.124342] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [14179.331556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14179.375065] alg: No test for adler32 (adler32-zlib) [14180.586557] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [14180.866502] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [14182.599782] Key type lgssc registered [14184.008704] Lustre: Echo OBD driver; http://www.lustre.org/ [14189.131712] LNet: Added LNI 192.168.202.151@tcp100 [8/256/0/180] [14190.394528] LNet: Added LNI 192.168.202.151@tcp101 [8/256/0/180] [14191.452587] LNet: Added LNI 192.168.202.151@tcp102 [8/256/0/180] [14192.576281] LNet: Added LNI 192.168.202.151@tcp103 [8/256/0/180] [14193.736408] LNet: Added LNI 192.168.202.151@tcp104 [8/256/0/180] [14194.644730] LNet: Added LNI 192.168.202.151@tcp105 [8/256/0/180] [14195.709267] LNet: Added LNI 192.168.202.151@tcp106 [8/256/0/180] [14196.687242] LNet: Added LNI 192.168.202.151@tcp107 [8/256/0/180] [14197.622302] LNet: Added LNI 192.168.202.151@tcp108 [8/256/0/180] [14198.907478] LNet: Added LNI 192.168.202.151@tcp109 [8/256/0/180] [14200.212812] LNet: Added LNI 192.168.202.151@tcp110 [8/256/0/180] [14201.571618] LNet: Added LNI 192.168.202.151@tcp111 [8/256/0/180] [14202.933815] LNet: Added LNI 192.168.202.151@tcp112 [8/256/0/180] [14204.157915] LNet: Added LNI 192.168.202.151@tcp113 [8/256/0/180] [14205.300152] LNet: Added LNI 192.168.202.151@tcp114 [8/256/0/180] [14206.328263] LNet: Added LNI 192.168.202.151@tcp115 [8/256/0/180] [14207.571846] LNet: Added LNI 192.168.202.151@tcp116 [8/256/0/180] [14208.848850] LNet: Added LNI 192.168.202.151@tcp117 [8/256/0/180] [14209.996611] LNet: Added LNI 192.168.202.151@tcp118 [8/256/0/180] [14211.310083] LNet: Added LNI 192.168.202.151@tcp119 [8/256/0/180] [14212.413794] LNet: Added LNI 192.168.202.151@tcp120 [8/256/0/180] [14213.388106] LNet: Added LNI 192.168.202.151@tcp121 [8/256/0/180] [14214.323955] LNet: Added LNI 192.168.202.151@tcp122 [8/256/0/180] [14215.392591] LNet: Added LNI 192.168.202.151@tcp123 [8/256/0/180] [14216.268773] LNet: Added LNI 192.168.202.151@tcp124 [8/256/0/180] [14217.470864] LNet: Added LNI 192.168.202.151@tcp125 [8/256/0/180] [14218.485450] LNet: Added LNI 192.168.202.151@tcp126 [8/256/0/180] [14219.725948] LNet: Added LNI 192.168.202.151@tcp127 [8/256/0/180] [14220.775756] LNet: Added LNI 192.168.202.151@tcp128 [8/256/0/180] [14221.897852] LNet: Added LNI 192.168.202.151@tcp129 [8/256/0/180] [14222.936192] LNet: Added LNI 192.168.202.151@tcp130 [8/256/0/180] [14223.827856] LNet: Added LNI 192.168.202.151@tcp131 [8/256/0/180] [14224.823970] LNet: Added LNI 192.168.202.151@tcp132 [8/256/0/180] [14225.767682] LNet: Added LNI 192.168.202.151@tcp133 [8/256/0/180] [14226.540402] LNet: Added LNI 192.168.202.151@tcp134 [8/256/0/180] [14227.601206] LNet: Added LNI 192.168.202.151@tcp135 [8/256/0/180] [14235.104865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [14235.138399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14236.289236] Lustre: 464938:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.151@tcp: can't add NID 192.168.202.151@tcp130: rc = -75 [14236.471613] LNetError: 462384:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [14236.495134] LNetError: Protocol error connecting to 192.168.202.151@tcp123 at host 192.168.202.151:988: is it running a compatible version of Lustre? [14236.515774] LNetError: Skipped 1 previous similar message [14236.594450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14236.622104] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14236.692305] Lustre: lustre-MDT0000: new disk, initializing [14236.821390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14236.852090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14237.216131] LNet: 1 local NIs in recovery (showing 1): 192.168.202.151@tcp123 [14237.218736] LNetError: 464962:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [14237.229501] LNetError: 464962:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [14237.233679] LNetError: Protocol error connecting to 192.168.202.151@tcp103 at host 192.168.202.151:988: is it running a compatible version of Lustre? [14239.265517] LNetError: 464962:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [14239.282146] LNetError: 464962:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [14239.288700] LNetError: Protocol error connecting to 192.168.202.151@tcp123 at host 192.168.202.151:988: is it running a compatible version of Lustre? [14239.300593] LNetError: Skipped 6 previous similar messages [14241.363704] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [14243.362634] LNetError: 462382:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [14243.375503] LNetError: Protocol error connecting to 192.168.202.151@tcp123 at host 192.168.202.151:988: is it running a compatible version of Lustre? [14251.553577] LNetError: 462385:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [14251.560710] LNetError: Protocol error connecting to 192.168.202.151@tcp123 at host 192.168.202.151:988: is it running a compatible version of Lustre? [14252.665359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14252.819357] Lustre: 464955:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14252.884685] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14252.888589] Lustre: Skipped 1 previous similar message [14253.002637] Lustre: lustre-MDT0001: new disk, initializing [14253.051750] Lustre: 465759:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.151@tcp: can't add NID 192.168.202.151@tcp130: rc = -75 [14253.063964] Lustre: 465759:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14253.198119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14253.245647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14253.254070] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14254.691179] Lustre: 465845:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.151@tcp: can't add NID 192.168.202.151@tcp130: rc = -75 [14254.698263] Lustre: 465845:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14258.293898] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [14266.034954] LNetError: 462383:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [14266.057256] LNetError: Protocol error connecting to 192.168.202.151@tcp108 at host 192.168.202.151:988: is it running a compatible version of Lustre? [14269.189863] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14275.954424] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14282.400876] LNetError: 462383:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [14282.417594] LNetError: 462383:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 12 previous similar messages [14282.426355] LNetError: Protocol error connecting to 192.168.202.151@tcp106 at host 192.168.202.151:988: is it running a compatible version of Lustre? [14282.434973] LNetError: Skipped 12 previous similar messages [14284.035440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14284.644321] Lustre: lustre-OST0000: new disk, initializing [14284.650266] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14284.664545] Lustre: 466933:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14284.902458] Lustre: 466938:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.151@tcp: can't add NID 192.168.202.151@tcp130: rc = -75 [14284.913830] Lustre: 466938:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14285.019701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14291.486831] Lustre: 467232:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.151@tcp: can't add NID 192.168.202.151@tcp130: rc = -75 [14291.505734] Lustre: 467232:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14291.562643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14291.570043] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14291.724412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14292.418520] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [14297.632747] LNet: 1 local NIs in recovery (showing 1): 192.168.202.151@tcp123 [14302.472863] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14312.419090] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14312.426275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14312.431847] LustreError: Skipped 1 previous similar message [14312.436836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14312.451794] Lustre: Skipped 1 previous similar message [14312.467035] Lustre: Skipped 1 previous similar message [14315.672231] Lustre: server umount lustre-OST0000 complete [14317.217402] LNetError: 464971:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [14317.232598] LNetError: 464971:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 14 previous similar messages [14317.239130] LNetError: Protocol error connecting to 192.168.202.151@tcp108 at host 192.168.202.151:988: is it running a compatible version of Lustre? [14317.251256] LNetError: Skipped 14 previous similar messages [14319.584802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14319.590707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14319.609384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14322.657101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14322.657413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14325.372432] Lustre: server umount lustre-MDT0000 complete [14329.330988] LustreError: 464946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790753651 with bad export cookie 6789106436350319765 [14329.335029] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14329.351056] LustreError: 464946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14329.741382] Lustre: server umount lustre-MDT0001 complete [14344.005530] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [14347.914068] Key type lgssc unregistered [14348.451643] LNet: 468847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14348.460538] LNetError: 468847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14348.467526] LNet: Removed LNI 192.168.202.151@tcp [14349.474532] LNet: Removed LNI 192.168.202.151@tcp100 [14349.485034] LNetError: 468847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14349.500775] LNetError: 468847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14349.519115] LNet: Removed LNI 192.168.202.151@tcp101 [14350.559955] LNet: Removed LNI 192.168.202.151@tcp102 [14350.562040] LNetError: 468847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14350.581260] LNetError: 468847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14350.591357] LNet: Removed LNI 192.168.202.151@tcp103 [14351.647482] LNet: Removed LNI 192.168.202.151@tcp104 [14352.672502] LNet: Removed LNI 192.168.202.151@tcp105 [14352.680846] LNetError: 468847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14352.699071] LNetError: 468847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14352.709609] LNet: Removed LNI 192.168.202.151@tcp106 [14353.759458] LNet: Removed LNI 192.168.202.151@tcp107 [14354.783182] LNet: Removed LNI 192.168.202.151@tcp108 [14355.807150] LNet: Removed LNI 192.168.202.151@tcp109 [14356.833310] LNet: Removed LNI 192.168.202.151@tcp110 [14356.852988] LNetError: 468847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14356.859852] LNetError: 468847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14356.864565] LNet: Removed LNI 192.168.202.151@tcp111 [14357.919413] LNet: Removed LNI 192.168.202.151@tcp112 [14358.944185] LNet: Removed LNI 192.168.202.151@tcp113 [14359.967363] LNet: Removed LNI 192.168.202.151@tcp114 [14360.992030] LNet: Removed LNI 192.168.202.151@tcp115 [14362.015154] LNet: Removed LNI 192.168.202.151@tcp116 [14363.042622] LNet: Removed LNI 192.168.202.151@tcp117 [14364.063360] LNet: Removed LNI 192.168.202.151@tcp118 [14365.087293] LNet: Removed LNI 192.168.202.151@tcp119 [14365.089823] LNetError: 468847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14365.098716] LNetError: 468847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14365.102936] LNet: Removed LNI 192.168.202.151@tcp120 [14366.112461] LNet: Removed LNI 192.168.202.151@tcp121 [14367.139146] LNet: Removed LNI 192.168.202.151@tcp122 [14367.146276] LNet: Removed LNI 192.168.202.151@tcp123 [14368.159425] LNet: Removed LNI 192.168.202.151@tcp124 [14369.183144] LNet: Removed LNI 192.168.202.151@tcp125 [14370.207367] LNet: Removed LNI 192.168.202.151@tcp126 [14371.231149] LNet: Removed LNI 192.168.202.151@tcp127 [14372.255596] LNet: Removed LNI 192.168.202.151@tcp128 [14372.263454] LNet: Removed LNI 192.168.202.151@tcp129 [14372.269888] LNet: Removed LNI 192.168.202.151@tcp130 [14372.278481] LNet: Removed LNI 192.168.202.151@tcp131 [14372.283547] LNet: Removed LNI 192.168.202.151@tcp132 [14372.294125] LNet: Removed LNI 192.168.202.151@tcp133 [14372.299380] LNet: Removed LNI 192.168.202.151@tcp134 [14372.310314] LNet: Removed LNI 192.168.202.151@tcp135 [14373.182968] Key type .llcrypt unregistered [14373.187460] Key type ._llcrypt unregistered [14400.361469] Key type ._llcrypt registered [14400.363073] Key type .llcrypt registered [14400.484183] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [14416.773833] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [14417.787600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14417.878358] alg: No test for adler32 (adler32-zlib) [14418.930867] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [14419.110212] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [14420.767293] Key type lgssc registered [14422.020756] Lustre: Echo OBD driver; http://www.lustre.org/ [14454.918385] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 03:36:15 (1790753775) [14473.845615] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [14477.039965] Key type lgssc unregistered [14477.344812] LNet: 472949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14477.349086] LNetError: 472949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14477.364869] LNet: Removed LNI 192.168.202.151@tcp [14478.309494] Key type .llcrypt unregistered [14478.312163] Key type ._llcrypt unregistered [14495.873480] Key type ._llcrypt registered [14495.876211] Key type .llcrypt registered [14495.963567] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [14496.920984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14497.038404] alg: No test for adler32 (adler32-zlib) [14498.149696] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [14498.350336] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [14500.063261] Key type lgssc registered [14501.314750] Lustre: Echo OBD driver; http://www.lustre.org/ [14511.006384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [14511.053405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14512.257730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14512.287352] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14512.356210] Lustre: lustre-MDT0000: new disk, initializing [14512.426882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14512.442107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14517.948055] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [14530.288894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14530.419361] Lustre: 474180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14530.508302] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14530.513807] Lustre: Skipped 1 previous similar message [14530.651408] Lustre: lustre-MDT0001: new disk, initializing [14530.817800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14530.873507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14530.887697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14536.099231] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [14546.438990] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14552.610878] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14559.190883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14559.431654] Lustre: lustre-OST0000: new disk, initializing [14559.437522] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14559.444138] Lustre: 476150:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14559.524556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14565.445623] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [14565.482978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14565.490596] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14565.692693] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14577.032957] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14582.477813] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing lnet_if_list [14589.528662] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14589.604223] LNet: Added LNI 192.168.202.151@tcp10 [8/256/0/180] [14594.230488] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14594.270149] LNet: Added LNI 192.168.202.151@tcp11 [8/256/0/180] [14597.528451] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14597.582251] LNet: Added LNI 192.168.202.151@tcp12 [8/256/0/180] [14602.005570] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14602.038959] LNet: Added LNI 192.168.202.151@tcp13 [8/256/0/180] [14606.175972] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14606.201686] LNet: Added LNI 192.168.202.151@tcp14 [8/256/0/180] [14610.183594] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14610.205785] LNet: Added LNI 192.168.202.151@tcp15 [8/256/0/180] [14614.594806] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14614.618367] LNet: Added LNI 192.168.202.151@tcp16 [8/256/0/180] [14619.253075] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14619.285724] LNet: Added LNI 192.168.202.151@tcp17 [8/256/0/180] [14623.724364] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14623.749678] LNet: Added LNI 192.168.202.151@tcp18 [8/256/0/180] [14628.305416] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14628.331791] LNet: Added LNI 192.168.202.151@tcp19 [8/256/0/180] [14633.337345] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14633.379300] LNet: Added LNI 192.168.202.151@tcp20 [8/256/0/180] [14639.359161] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14639.405653] LNet: Added LNI 192.168.202.151@tcp21 [8/256/0/180] [14645.297715] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14645.335505] LNet: Added LNI 192.168.202.151@tcp22 [8/256/0/180] [14652.417788] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14652.482497] LNet: Added LNI 192.168.202.151@tcp23 [8/256/0/180] [14658.501457] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14658.528244] LNet: Added LNI 192.168.202.151@tcp24 [8/256/0/180] [14665.713393] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14665.809155] LNet: Added LNI 192.168.202.151@tcp25 [8/256/0/180] [14672.452541] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14672.512308] LNet: Added LNI 192.168.202.151@tcp26 [8/256/0/180] [14680.841715] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14680.872645] LNet: Added LNI 192.168.202.151@tcp27 [8/256/0/180] [14687.286213] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14687.306091] LNet: Added LNI 192.168.202.151@tcp28 [8/256/0/180] [14694.646853] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14694.670619] LNet: Added LNI 192.168.202.151@tcp29 [8/256/0/180] [14700.878924] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14700.906458] LNet: Added LNI 192.168.202.151@tcp30 [8/256/0/180] [14707.868118] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14707.912095] LNet: Added LNI 192.168.202.151@tcp31 [8/256/0/180] [14715.833700] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14715.860626] LNet: Added LNI 192.168.202.151@tcp32 [8/256/0/180] [14723.539931] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14723.619517] LNet: Added LNI 192.168.202.151@tcp33 [8/256/0/180] [14731.463434] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14731.490076] LNet: Added LNI 192.168.202.151@tcp34 [8/256/0/180] [14738.859826] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14738.903050] LNet: Added LNI 192.168.202.151@tcp35 [8/256/0/180] [14745.801326] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14745.839630] LNet: Added LNI 192.168.202.151@tcp36 [8/256/0/180] [14753.298506] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14753.327358] LNet: Added LNI 192.168.202.151@tcp37 [8/256/0/180] [14760.303277] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14760.332338] LNet: Added LNI 192.168.202.151@tcp38 [8/256/0/180] [14766.331359] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14766.378170] LNet: Added LNI 192.168.202.151@tcp39 [8/256/0/180] [14773.550968] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14773.614274] LNet: Added LNI 192.168.202.151@tcp40 [8/256/0/180] [14781.020587] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14781.065644] LNet: Added LNI 192.168.202.151@tcp41 [8/256/0/180] [14788.626902] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14788.655567] LNet: Added LNI 192.168.202.151@tcp42 [8/256/0/180] [14797.180186] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14797.210910] LNet: Added LNI 192.168.202.151@tcp43 [8/256/0/180] [14806.488075] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14806.537718] LNet: Added LNI 192.168.202.151@tcp44 [8/256/0/180] [14816.306523] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14816.327587] LNet: Added LNI 192.168.202.151@tcp45 [8/256/0/180] [14825.560482] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14825.617872] LNet: Added LNI 192.168.202.151@tcp46 [8/256/0/180] [14834.532587] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14834.548819] LNet: Added LNI 192.168.202.151@tcp47 [8/256/0/180] [14844.798059] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14844.841550] LNet: Added LNI 192.168.202.151@tcp48 [8/256/0/180] [14853.361436] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14853.389940] LNet: Added LNI 192.168.202.151@tcp49 [8/256/0/180] [14864.148989] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14864.192869] LNet: Added LNI 192.168.202.151@tcp50 [8/256/0/180] [14888.419835] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14888.427829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14888.438455] Lustre: Skipped 2 previous similar messages [14888.452317] Lustre: Skipped 2 previous similar messages [14891.593422] Lustre: server umount lustre-MDT0000 complete [14895.112677] LustreError: 474171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790754217 with bad export cookie 11473930973409963548 [14895.120137] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14895.126165] LustreError: 474171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14895.558635] Lustre: server umount lustre-MDT0001 complete [14913.260220] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [14915.041969] Lustre: 473441:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790754220/real 1790754220] req@00000000d680b368 x1877741477738368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790754236 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14915.079500] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14916.130530] Lustre: 473441:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790754222/real 1790754222] req@00000000938010fd x1877741477738496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790754238 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14919.139955] Lustre: 473444:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790754225/real 1790754225] req@00000000d7154780 x1877741477738624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790754241 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14921.250589] Lustre: 473444:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790754227/real 1790754227] req@0000000038117c48 x1877741477738880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790754243 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14922.090861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14926.303219] Lustre: 473442:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790754232/real 1790754232] req@000000009f243aed x1877741477739008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790754248 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14940.933190] LustreError: 476144:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14940.946404] LustreError: 476144:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [14940.978365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14944.045960] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14944.048411] LustreError: 476145:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14944.865377] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [14945.062389] LustreError: 476144:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14950.373792] LustreError: 476145:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14954.523564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14954.936930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14959.833751] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [14961.151246] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14967.183514] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14972.990958] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14992.352259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14992.363679] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14992.389722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14995.943382] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14995.947888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14995.960118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14998.399090] Lustre: server umount lustre-OST0000 complete [15006.178562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15006.194990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15006.242430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15006.262570] Lustre: Skipped 2 previous similar messages [15009.132967] Lustre: server umount lustre-MDT0000 complete [15012.321100] LustreError: 500394:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15012.347181] LustreError: 500394:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15013.761241] LustreError: 474171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790754335 with bad export cookie 11473930973409966173 [15013.797701] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15014.450265] Lustre: server umount lustre-MDT0001 complete [15026.508718] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [15029.451479] Key type lgssc unregistered [15029.940607] LNet: 503171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15029.948748] LNetError: 503171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15029.954058] LNet: Removed LNI 192.168.202.151@tcp [15029.957229] LNet: Removed LNI 192.168.202.151@tcp10 [15029.959618] LNet: Removed LNI 192.168.202.151@tcp11 [15029.961129] LNet: Removed LNI 192.168.202.151@tcp12 [15029.962945] LNet: Removed LNI 192.168.202.151@tcp13 [15029.965642] LNet: Removed LNI 192.168.202.151@tcp14 [15029.968413] LNet: Removed LNI 192.168.202.151@tcp15 [15029.971193] LNet: Removed LNI 192.168.202.151@tcp16 [15029.973251] LNet: Removed LNI 192.168.202.151@tcp17 [15029.976270] LNet: Removed LNI 192.168.202.151@tcp18 [15029.978554] LNet: Removed LNI 192.168.202.151@tcp19 [15029.981321] LNet: Removed LNI 192.168.202.151@tcp20 [15029.984285] LNet: Removed LNI 192.168.202.151@tcp21 [15029.986650] LNet: Removed LNI 192.168.202.151@tcp22 [15029.989803] LNet: Removed LNI 192.168.202.151@tcp23 [15029.991617] LNet: Removed LNI 192.168.202.151@tcp24 [15029.993882] LNet: Removed LNI 192.168.202.151@tcp25 [15029.996426] LNet: Removed LNI 192.168.202.151@tcp26 [15029.998962] LNet: Removed LNI 192.168.202.151@tcp27 [15030.001465] LNet: Removed LNI 192.168.202.151@tcp28 [15030.004343] LNet: Removed LNI 192.168.202.151@tcp29 [15030.006496] LNet: Removed LNI 192.168.202.151@tcp30 [15030.009604] LNet: Removed LNI 192.168.202.151@tcp31 [15030.017495] LNet: Removed LNI 192.168.202.151@tcp32 [15030.022992] LNet: Removed LNI 192.168.202.151@tcp33 [15030.025100] LNet: Removed LNI 192.168.202.151@tcp34 [15030.036776] LNet: Removed LNI 192.168.202.151@tcp35 [15030.041512] LNet: Removed LNI 192.168.202.151@tcp36 [15030.045224] LNet: Removed LNI 192.168.202.151@tcp37 [15030.051614] LNet: Removed LNI 192.168.202.151@tcp38 [15030.054968] LNet: Removed LNI 192.168.202.151@tcp39 [15030.062415] LNet: Removed LNI 192.168.202.151@tcp40 [15030.069066] LNet: Removed LNI 192.168.202.151@tcp41 [15030.078822] LNet: Removed LNI 192.168.202.151@tcp42 [15030.084615] LNet: Removed LNI 192.168.202.151@tcp43 [15030.094279] LNet: Removed LNI 192.168.202.151@tcp44 [15030.096407] LNet: Removed LNI 192.168.202.151@tcp45 [15030.104235] LNet: Removed LNI 192.168.202.151@tcp46 [15030.106837] LNet: Removed LNI 192.168.202.151@tcp47 [15030.112482] LNet: Removed LNI 192.168.202.151@tcp48 [15030.115305] LNet: Removed LNI 192.168.202.151@tcp49 [15030.135928] LNet: Removed LNI 192.168.202.151@tcp50 [15030.772961] Key type .llcrypt unregistered [15030.774925] Key type ._llcrypt unregistered [15050.641486] Key type ._llcrypt registered [15050.645733] Key type .llcrypt registered [15050.750668] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [15065.040806] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [15065.979949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15066.086516] alg: No test for adler32 (adler32-zlib) [15067.115594] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [15067.324736] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [15068.975246] Key type lgssc registered [15070.061929] Lustre: Echo OBD driver; http://www.lustre.org/ [15101.544210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [15101.565403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15102.799505] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15102.831560] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15102.934669] Lustre: lustre-MDT0000: new disk, initializing [15102.994849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15103.011848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15106.692611] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15116.175812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15116.295916] Lustre: 506381:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15116.343636] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15116.347315] Lustre: Skipped 1 previous similar message [15116.405401] Lustre: lustre-MDT0001: new disk, initializing [15116.485902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15116.517661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15116.529429] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15120.755126] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15130.116538] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15134.681901] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15142.236607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15142.441365] Lustre: *** cfs_fail_loc=90f, val=0*** [15142.514674] Lustre: lustre-OST0000: new disk, initializing [15142.517415] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15142.521815] Lustre: 508401:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15142.616017] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15148.623992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15148.628079] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15148.734301] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15149.910914] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15159.477592] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15169.512856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15169.533943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15169.534721] LustreError: Skipped 1 previous similar message [15169.560255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15169.566160] Lustre: Skipped 1 previous similar message [15169.598506] Lustre: Skipped 1 previous similar message [15173.655518] Lustre: server umount lustre-OST0000 complete [15178.209796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15178.217882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15178.249108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15179.746117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15179.748620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15183.750908] Lustre: server umount lustre-MDT0000 complete [15187.326769] LustreError: 508398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790754509 with bad export cookie 9998222683742555946 [15187.328428] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15187.338693] LustreError: 508398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15187.687836] Lustre: server umount lustre-MDT0001 complete [15197.840239] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [15200.432961] Key type lgssc unregistered [15200.722544] LNet: 510303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15200.758358] LNetError: 510303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15200.781766] LNet: Removed LNI 192.168.202.151@tcp [15201.951359] Key type .llcrypt unregistered [15201.953624] Key type ._llcrypt unregistered [15221.537870] Key type ._llcrypt registered [15221.539765] Key type .llcrypt registered [15221.623514] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [15222.844133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15222.864116] alg: No test for adler32 (adler32-zlib) [15224.010284] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [15224.229170] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [15225.887256] Key type lgssc registered [15226.833763] Lustre: Echo OBD driver; http://www.lustre.org/ [15234.011313] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [15234.025300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15235.442385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15239.226371] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15245.793792] LustreError: 511590:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15245.804834] LustreError: 511590:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [15247.752111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15248.135626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15252.281276] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15259.467831] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15264.455803] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15270.087748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15270.466201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15275.796125] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15282.124847] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15291.951517] loop0: detected capacity change from 0 to 204800000 [15292.132726] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15292.140144] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15292.148805] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15297.725087] loop0: detected capacity change from 0 to 204800000 [15297.941213] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15297.954854] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15297.974415] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15302.886332] loop0: detected capacity change from 0 to 204800000 [15308.676890] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15308.887685] Lustre: fs15246-OST0000: new disk, initializing [15308.895466] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15308.906101] Lustre: 515183:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15309.010137] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15314.597882] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15325.264732] loop1: detected capacity change from 0 to 204800000 [15331.120555] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15331.279858] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15331.350412] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15331.404304] Lustre: fs15246-MDT0000: new disk, initializing [15331.566239] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15331.588179] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15331.693343] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15332.704025] Lustre: 516304:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000041bfcf9 x1877742238871680/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790754670 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15335.931187] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15341.807043] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15341.818824] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15341.835387] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15355.381253] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15355.412212] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15360.490300] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15361.012496] Lustre: server umount fs15246-MDT0000 complete [15365.608320] LustreError: 511586:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15370.727028] LustreError: 512832:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15371.332749] Lustre: server umount fs15246-OST0000 complete [15385.066726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15385.082510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15385.115820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15386.087983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15390.683836] Lustre: server umount lustre-OST0000 complete [15396.323134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15396.326947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15396.330855] Lustre: Skipped 1 previous similar message [15396.345128] Lustre: Skipped 3 previous similar messages [15400.541063] Lustre: server umount lustre-MDT0000 complete [15401.444799] LustreError: 511590:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15401.465305] LustreError: 511590:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15404.128553] LustreError: 517442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790754726 with bad export cookie 3335802301453455798 [15404.135861] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15404.139256] LustreError: 517442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15404.376107] Lustre: server umount lustre-MDT0001 complete [15414.798033] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [15417.349492] Key type lgssc unregistered [15417.600069] LNet: 519398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15417.607388] LNetError: 519398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15417.621841] LNet: Removed LNI 192.168.202.151@tcp [15418.416197] Key type .llcrypt unregistered [15418.422754] Key type ._llcrypt unregistered [15435.515796] Key type ._llcrypt registered [15435.517438] Key type .llcrypt registered [15435.592867] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [15437.233578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15437.298133] alg: No test for adler32 (adler32-zlib) [15438.403813] Lustre: Lustre: Build Version: 2.17.59_3_g94a7b9b [15438.616372] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [15440.343269] Key type lgssc registered [15442.166446] Lustre: Echo OBD driver; http://www.lustre.org/ [15476.810028] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [15485.137218] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [15513.230304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [15513.259330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15514.443625] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15514.462599] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15514.535921] Lustre: lustre-MDT0000: new disk, initializing [15514.605898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15514.623715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15518.404947] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15528.607830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15528.756488] Lustre: 523878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15528.806682] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15528.809814] Lustre: Skipped 1 previous similar message [15528.886287] Lustre: lustre-MDT0001: new disk, initializing [15528.969546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15528.996715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15529.006958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15533.860688] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15543.214801] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15548.387393] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15554.973061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15555.211428] Lustre: lustre-OST0000: new disk, initializing [15555.214882] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15555.220597] Lustre: 525850:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15555.281038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15561.703696] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15565.427428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15565.441023] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15565.496813] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15569.718991] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15573.749947] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15573.889725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15578.102252] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15578.288803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15581.157258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15581.161084] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15581.168400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15581.174342] Lustre: Skipped 1 previous similar message [15581.192903] LustreError: Skipped 1 previous similar message [15586.344720] Lustre: server umount lustre-OST0000 complete [15590.370193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15590.375934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15590.390237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15590.397949] Lustre: Skipped 1 previous similar message [15591.395520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15591.395918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15591.416600] Lustre: Skipped 1 previous similar message [15596.514826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15596.526398] Lustre: Skipped 1 previous similar message [15601.633587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15601.639088] Lustre: Skipped 1 previous similar message [15604.191139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15604.399180] Lustre: server umount lustre-MDT0000 complete [15606.760878] LustreError: 523887:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15606.784195] LustreError: 523887:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [15607.543240] LustreError: 524685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790754929 with bad export cookie 5567036618304437054 [15607.548888] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15607.552289] LustreError: 524685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15607.798310] Lustre: server umount lustre-MDT0001 complete [15628.690858] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [15638.849838] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [15664.396400] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15390 sec ========= 03:56:24 (1790754984) [15665.618826] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:56:25 (1790754985) === [15669.009362] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:56:28 (1790754988) === [15713.647794] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [15718.538269] Key type lgssc unregistered [15719.190583] LNet: 532494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15719.211261] LNetError: 532494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15719.238355] LNet: Removed LNI 192.168.202.151@tcp [15720.919115] Key type .llcrypt unregistered [15720.920481] Key type ._llcrypt unregistered