[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 532531956 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001023] APIC: Switch to symmetric I/O mode setup [ 0.002408] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007679] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010133] LSM: Security Framework initializing [ 0.011047] Yama: becoming mindful. [ 0.012035] SELinux: Initializing. [ 0.013064] *** VALIDATE selinux *** [ 0.022436] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027267] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028135] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029098] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031003] *** VALIDATE tmpfs *** [ 0.032428] *** VALIDATE proc *** [ 0.033226] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.036153] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039030] Spectre V2 : User space: Vulnerable [ 0.040006] Speculative Store Bypass: Vulnerable [ 0.043372] debug: unmapping init [mem 0xffffffffb6259000-0xffffffffb6260fff] [ 0.046123] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047626] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048022] ... version: 2 [ 0.049013] ... bit width: 48 [ 0.050011] ... generic registers: 4 [ 0.051008] ... value mask: 0000ffffffffffff [ 0.052010] ... max period: 00007fffffffffff [ 0.053014] ... fixed-purpose events: 3 [ 0.054008] ... event mask: 000000070000000f [ 0.055288] rcu: Hierarchical SRCU implementation. [ 0.057410] smp: Bringing up secondary CPUs ... [ 0.058550] x86: Booting SMP configuration: [ 0.059022] .... node #0, CPUs: #1 #2 #3 [ 0.066358] smp: Brought up 1 node, 4 CPUs [ 0.068012] smpboot: Max logical packages: 1 [ 0.069013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.112073] node 0 deferred pages initialised in 40ms [ 0.115315] devtmpfs: initialized [ 0.116280] x86/mm: Memory block size: 128MB [ 0.119690] gcov: version magic: 0x41383552 [ 0.123246] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.124075] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.125265] pinctrl core: initialized pinctrl subsystem [ 0.126174] [ 0.126875] ************************************************************* [ 0.127013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.128012] ** ** [ 0.129012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.130013] ** ** [ 0.131014] ** This means that this kernel is built to expose internal ** [ 0.132013] ** IOMMU data structures, which may compromise security on ** [ 0.133012] ** your system. ** [ 0.134017] ** ** [ 0.135011] ** If you see this message and you are not debugging the ** [ 0.136014] ** kernel, report this immediately to your vendor! ** [ 0.137016] ** ** [ 0.138013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.139016] ************************************************************* [ 0.140776] NET: Registered protocol family 16 [ 0.142136] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.145062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.148067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.152120] cpuidle: using governor menu [ 0.155031] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.157597] PCI: Using configuration type 1 for base access [ 0.160126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.170059] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.172087] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.176099] cryptd: max_cpu_qlen set to 1000 [ 0.180295] ACPI: Added _OSI(Module Device) [ 0.182017] ACPI: Added _OSI(Processor Device) [ 0.183010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.185014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.190459] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.197349] ACPI: Interpreter enabled [ 0.199057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.200012] ACPI: Using IOAPIC for interrupt routing [ 0.202132] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.206426] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.219324] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.220040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.222018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.226073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.231372] acpiphp: Slot [2] registered [ 0.233197] acpiphp: Slot [5] registered [ 0.234117] acpiphp: Slot [6] registered [ 0.236125] acpiphp: Slot [7] registered [ 0.238115] acpiphp: Slot [8] registered [ 0.239113] acpiphp: Slot [9] registered [ 0.241114] acpiphp: Slot [10] registered [ 0.243141] acpiphp: Slot [3] registered [ 0.244103] acpiphp: Slot [4] registered [ 0.246109] acpiphp: Slot [11] registered [ 0.248081] acpiphp: Slot [12] registered [ 0.250107] acpiphp: Slot [13] registered [ 0.251099] acpiphp: Slot [14] registered [ 0.253105] acpiphp: Slot [15] registered [ 0.254092] acpiphp: Slot [16] registered [ 0.256080] acpiphp: Slot [17] registered [ 0.258106] acpiphp: Slot [18] registered [ 0.259166] acpiphp: Slot [19] registered [ 0.261124] acpiphp: Slot [20] registered [ 0.263105] acpiphp: Slot [21] registered [ 0.264113] acpiphp: Slot [22] registered [ 0.266111] acpiphp: Slot [23] registered [ 0.268109] acpiphp: Slot [24] registered [ 0.269093] acpiphp: Slot [25] registered [ 0.271097] acpiphp: Slot [26] registered [ 0.273095] acpiphp: Slot [27] registered [ 0.274107] acpiphp: Slot [28] registered [ 0.276096] acpiphp: Slot [29] registered [ 0.278057] acpiphp: Slot [30] registered [ 0.279103] acpiphp: Slot [31] registered [ 0.281058] PCI host bridge to bus 0000:00 [ 0.282016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.285021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.288024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.290020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.293021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.296024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.298175] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.301011] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.305389] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.314842] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.320503] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.323019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.325013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.327019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.328450] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.330058] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.332041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.334751] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.340015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.356019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.361014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.368126] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.380014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.386016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.400016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.412250] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.420018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.428019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.449020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.458875] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.465015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.471017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.485017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.499696] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.511031] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.521022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.546017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.559098] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.570018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.580015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.607018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.619423] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.626014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.633016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.648014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.657835] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.659296] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.661530] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.663217] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.665153] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.671124] iommu: Default domain type: Passthrough [ 0.672000] SCSI subsystem initialized [ 0.672000] ACPI: bus type USB registered [ 0.674091] usbcore: registered new interface driver usbfs [ 0.676062] usbcore: registered new interface driver hub [ 0.677043] usbcore: registered new device driver usb [ 0.678127] pps_core: LinuxPPS API ver. 1 registered [ 0.680009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.683046] PTP clock support registered [ 0.684211] EDAC MC: Ver: 3.0.0 [ 0.685816] PCI: Using ACPI for IRQ routing [ 0.686000] NetLabel: Initializing [ 0.686000] NetLabel: domain hash size = 128 [ 0.688010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.690071] NetLabel: unlabeled traffic allowed by default [ 0.692037] vgaarb: loaded [ 0.693398] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.694008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.698849] clocksource: Switched to clocksource kvm-clock [ 0.808422] VFS: Disk quotas dquot_6.6.0 [ 0.809916] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.813919] *** VALIDATE ramfs *** [ 0.815831] *** VALIDATE hugetlbfs *** [ 0.816941] pnp: PnP ACPI init [ 0.819270] pnp: PnP ACPI: found 6 devices [ 0.833903] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.837108] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.839396] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.841739] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.844131] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.846439] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.849114] NET: Registered protocol family 2 [ 0.851500] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.856604] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.859329] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.863228] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.866265] TCP: Hash tables configured (established 65536 bind 65536) [ 0.869300] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.872549] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.875483] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.878654] NET: Registered protocol family 1 [ 0.881261] RPC: Registered named UNIX socket transport module. [ 0.883518] RPC: Registered udp transport module. [ 0.885367] RPC: Registered tcp transport module. [ 0.887217] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.889636] NET: Registered protocol family 44 [ 0.891224] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.893167] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.894738] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.896742] PCI: CLS 0 bytes, default 64 [ 0.898924] Unpacking initramfs... [ 2.453672] debug: unmapping init [mem 0xffff9e0abcc54000-0xffff9e0abffbffff] [ 2.457949] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.460430] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.463083] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.957747] Initialise system trusted keyrings [ 2.959811] Key type blacklist registered [ 2.961907] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.968889] zbud: loaded [ 2.972325] *** VALIDATE nfs *** [ 2.973430] *** VALIDATE nfs4 *** [ 2.974916] pstore: using deflate compression [ 2.978307] Platform Keyring initialized [ 3.065789] NET: Registered protocol family 38 [ 3.067326] Key type asymmetric registered [ 3.068359] Asymmetric key parser 'x509' registered [ 3.069600] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.072268] io scheduler mq-deadline registered [ 3.074070] io scheduler kyber registered [ 3.075765] io scheduler bfq registered [ 3.077422] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.080634] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.083621] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.086730] ACPI: Power Button [PWRF] [ 3.092312] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.098835] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.115220] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.124074] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.139723] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.167722] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.196636] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.201212] Non-volatile memory driver v1.3 [ 3.202345] Linux agpgart interface v0.103 [ 3.227581] virtio_blk virtio1: [vda] 146200 512-byte logical blocks (74.9 MB/71.4 MiB) [ 3.230476] vda: detected capacity change from 0 to 74854400 [ 3.242989] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.246973] vdb: detected capacity change from 0 to 1073741824 [ 3.269412] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.271200] vdc: detected capacity change from 0 to 2621440000 [ 3.296683] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.300577] vdd: detected capacity change from 0 to 2621440000 [ 3.320420] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.322309] vde: detected capacity change from 0 to 4294967296 [ 3.336125] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.338335] vdf: detected capacity change from 0 to 4294967296 [ 3.346743] libphy: Fixed MDIO Bus: probed [ 3.350775] usbcore: registered new interface driver usbserial_generic [ 3.352517] usbserial: USB Serial support registered for generic [ 3.354056] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.356653] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.358019] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.359536] mousedev: PS/2 mouse device common for all mice [ 3.361702] rtc_cmos 00:05: RTC can wake from S4 [ 3.365046] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.368272] rtc_cmos 00:05: registered as rtc0 [ 3.371878] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.372245] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.375231] intel_pstate: CPU model not supported [ 3.380667] hid: raw HID events driver (C) Jiri Kosina [ 3.381582] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.382739] usbcore: registered new interface driver usbhid [ 3.386289] usbhid: USB HID core driver [ 3.387948] drop_monitor: Initializing network drop monitor service [ 3.390792] Initializing XFRM netlink socket [ 3.393237] NET: Registered protocol family 10 [ 3.396446] Segment Routing with IPv6 [ 3.397783] NET: Registered protocol family 17 [ 3.399330] mpls_gso: MPLS GSO support [ 3.404350] RAS: Correctable Errors collector initialized. [ 3.406637] AVX version of gcm_enc/dec engaged. [ 3.408815] AES CTR mode by8 optimization enabled [ 3.474551] sched_clock: Marking stable (3474468808, 0)->(4393275997, -918807189) [ 3.477625] registered taskstats version 1 [ 3.479223] Loading compiled-in X.509 certificates [ 3.481555] zswap: loaded using pool lzo/zbud [ 3.503381] Key type big_key registered [ 3.516117] Key type encrypted registered [ 3.518178] ima: No TPM chip found, activating TPM-bypass! [ 3.520570] ima: Allocated hash algorithm: sha1 [ 3.521515] ima: No architecture policies found [ 3.522595] evm: Initialising EVM extended attributes: [ 3.523725] evm: security.selinux [ 3.524515] evm: security.ima [ 3.525195] evm: security.capability [ 3.525951] evm: HMAC attrs: 0x1 [ 3.527585] rtc_cmos 00:05: setting system clock to 2026-08-26 08:39:49 UTC (1787733589) [ 3.531464] debug: unmapping init [mem 0xffffffffb7203000-0xffffffffb73fffff] [ 3.533442] debug: unmapping init [mem 0xffffffffb5f82000-0xffffffffb6258fff] [ 3.541096] Write protecting the kernel read-only data: 28672k [ 3.543753] debug: unmapping init [mem 0xffffffffb4603000-0xffffffffb47fffff] [ 3.545587] debug: unmapping init [mem 0xffffffffb4f14000-0xffffffffb4ffffff] [ 3.569054] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.574792] systemd[1]: Detected virtualization kvm. [ 3.576296] systemd[1]: Detected architecture x86-64. [ 3.577660] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.601135] systemd[1]: No hostname configured. [ 3.602291] systemd[1]: Set hostname to . [ 3.603623] random: systemd: uninitialized urandom read (16 bytes read) [ 3.605138] systemd[1]: Initializing machine ID from random generator. [ 3.659140] random: ln: uninitialized urandom read (6 bytes read) [ 3.723760] random: systemd: uninitialized urandom read (16 bytes read) [ 3.727314] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.736614] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.740817] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.324232] device-mapper: uevent: version 1.0.3 [ 4.325620] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.974818] virtio_net virtio0 ens2: renamed from eth0 [ 5.002284] random: fast init done [ 5.028194] scsi host0: ata_piix [ 5.052888] scsi host1: ata_piix [ 5.054099] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.055925] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.031481] random: crng init done [ 10.034434] random: 7 urandom warning(s) missed due to ratelimiting [ 10.039872] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.217401] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.460548] printk: systemd: 26 output lines suppressed due to ratelimiting [ 13.998167] SELinux: Disabled at runtime. [ 14.087395] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.106405] systemd[1]: Detected virtualization kvm. [ 14.107566] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.029569] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.036699] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.045487] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.050471] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.055824] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.066405] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.084171] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ 15.356572] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 15.941557] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 16.426370] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 16.520227] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 16.758168] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 16.802484] EDAC sbridge: Ver: 1.1.2 [ 19.634875] Key type dns_resolver registered [ 20.281991] NFS: Registering the id_resolver key type [ 20.287635] Key type id_resolver registered [ 20.293024] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg137-server login: [ 30.621051] hrtimer: interrupt took 2064758 ns [ 75.618251] libcfs: loading out-of-tree module taints kernel. [ 75.673939] Key type ._llcrypt registered [ 75.676946] Key type .llcrypt registered [ 75.794501] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 92.982557] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 94.523686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 94.540976] alg: No test for adler32 (adler32-zlib) [ 95.977247] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 96.854614] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 98.583262] Key type lgssc registered [ 100.095158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 117.656234] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 152.008590] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 162.714198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 162.753692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 163.968359] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 164.001526] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 164.079441] Lustre: lustre-MDT0000: new disk, initializing [ 164.123215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 164.138839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 167.320141] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.381332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 178.504509] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 178.530808] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 178.535354] Lustre: Skipped 1 previous similar message [ 178.607717] Lustre: lustre-MDT0001: new disk, initializing [ 178.675719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 178.729853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 178.747626] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 182.699656] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 186.741706] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 194.302678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 194.501436] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 194.510665] Lustre: lustre-OST0000: new disk, initializing [ 194.510980] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 194.511515] Lustre: 8446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 194.569605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.690583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 199.706873] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 199.731684] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 199.744790] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.791645] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 212.938974] Lustre: lustre-OST0001: new disk, initializing [ 212.942370] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 212.944769] Lustre: 9519:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 212.995082] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 219.422432] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.226799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 220.245737] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 220.364569] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 231.889615] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 238.441439] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 245.401471] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing check_logdir /tmp/testlogs/ [ 250.413106] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing yml_node [ 254.453785] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 256.900248] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 259.339447] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 260.725581] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 26 04:44:05 EDT 2026 [ 275.164502] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 276.585705] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 292.320131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 292.320547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 292.338625] Lustre: Skipped 2 previous similar messages [ 292.358211] Lustre: Skipped 3 previous similar messages [ 297.447991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 297.950898] Lustre: server umount lustre-MDT0000 complete [ 301.780156] LustreError: 9518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787733887 with bad export cookie 15870492972419303438 [ 301.790258] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 301.804298] LustreError: 9518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 302.560555] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 302.580475] Lustre: Skipped 1 previous similar message [ 302.586453] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 302.589415] Lustre: Skipped 4 previous similar messages [ 306.592868] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 306.606087] Lustre: Skipped 1 previous similar message [ 311.722728] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 311.753085] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 316.383111] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 316.737493] Lustre: server umount lustre-MDT0001 complete [ 320.895187] Lustre: server umount lustre-OST0000 complete [ 325.369503] Lustre: server umount lustre-OST0001 complete [ 332.969763] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 342.414546] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 375.342540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 375.557895] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 375.597925] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 375.690155] Lustre: lustre-MDT0000: new disk, initializing [ 375.798297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 375.834882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 379.799118] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 389.107344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 389.237346] Lustre: 15071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 389.295922] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 389.299732] Lustre: Skipped 1 previous similar message [ 389.364572] Lustre: lustre-MDT0001: new disk, initializing [ 389.437134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 389.478622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 389.492371] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 394.444549] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 402.887506] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 407.598938] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 411.844711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 412.015329] Lustre: lustre-OST0000: new disk, initializing [ 412.018735] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 412.023371] Lustre: 17039:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 412.084940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 413.931761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 413.944770] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 414.007619] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 416.127746] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 423.367426] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 427.715262] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 427.965045] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 432.058626] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 432.236261] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 434.658571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 434.659344] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 434.660315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 434.660321] Lustre: Skipped 2 previous similar messages [ 434.665650] Lustre: Skipped 2 previous similar messages [ 434.669881] LustreError: Skipped 1 previous similar message [ 439.783234] LustreError: 17025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 439.792974] LustreError: 17025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 439.855540] Lustre: server umount lustre-OST0000 complete [ 444.901266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 449.411117] Lustre: server umount lustre-MDT0000 complete [ 450.017611] LustreError: 15084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 450.017611] LustreError: 15079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 452.821704] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787734038 with bad export cookie 15870492972419304964 [ 452.826989] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 452.830127] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 453.132922] Lustre: server umount lustre-MDT0001 complete [ 456.247984] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 04:47:21 (1787734041) [ 458.123749] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 459.486754] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 04:47:24 (1787734044) [ 464.988453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.306612] LustreError: 19446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 465.388471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.195580] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 470.498515] LustreError: 19447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 478.391568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 483.390924] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 489.568914] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 494.405270] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 500.028453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 505.764822] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 512.205334] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 518.630371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 518.639202] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 518.653117] Lustre: Skipped 1 previous similar message [ 518.659560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 518.664925] Lustre: Skipped 3 previous similar messages [ 520.673208] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 524.562804] Lustre: server umount lustre-OST0000 complete [ 529.888951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 529.898742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 529.918387] Lustre: Skipped 1 previous similar message [ 534.526393] Lustre: server umount lustre-MDT0000 complete [ 536.038017] LustreError: 20174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.062153] LustreError: 20174:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 537.557597] LustreError: 19428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787734123 with bad export cookie 15870492972419305846 [ 537.573576] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 537.575591] LustreError: 19428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 537.988266] Lustre: server umount lustre-MDT0001 complete [ 547.463469] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 549.948413] Key type lgssc unregistered [ 550.318406] LNet: 22981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 550.330577] LNetError: 22981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 550.354672] LNet: Removed LNI 192.168.201.137@tcp [ 551.218126] Key type .llcrypt unregistered [ 551.219506] Key type ._llcrypt unregistered [ 572.414808] Key type ._llcrypt registered [ 572.416591] Key type .llcrypt registered [ 572.501499] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 573.486049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 573.512442] alg: No test for adler32 (adler32-zlib) [ 574.679519] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 575.067341] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 576.839183] Key type lgssc registered [ 578.293430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 585.780093] loop: module loaded [ 585.850296] loop0: detected capacity change from 0 to 204800000 [ 585.996901] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 586.011247] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 586.237371] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 590.469638] loop0: detected capacity change from 0 to 204800000 [ 590.724548] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 590.745679] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 590.775021] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 596.101896] loop0: detected capacity change from 0 to 204800000 [ 600.374880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 600.418736] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.692091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 601.767183] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 601.870770] Lustre: lustre-MDT0000: new disk, initializing [ 601.997237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 602.028350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 605.918409] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 614.744839] loop1: detected capacity change from 0 to 204800000 [ 618.669079] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 618.902941] Lustre: lustre-OST07c6: new disk, initializing [ 618.906383] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 618.909598] Lustre: Skipped 1 previous similar message [ 618.913781] Lustre: 26165:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 618.984134] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 619.244772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 619.252390] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 619.286550] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 624.099414] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 628.607651] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 628.619206] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 628.623177] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 628.627841] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 628.633524] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 628.636935] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 647.084458] Lustre: 26770:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 647.090226] Lustre: 26770:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 647.094510] Lustre: 26770:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 647.099544] Lustre: 26770:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 647.104483] Lustre: 26770:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 647.109225] Lustre: 26770:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 651.565042] Lustre: 26770:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 651.582103] Lustre: 26770:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 651.594510] Lustre: 26770:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 651.607094] Lustre: 26770:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 651.627806] Lustre: 26770:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 651.642431] Lustre: 26770:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 651.655740] Lustre: 26770:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 651.667606] Lustre: 26770:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 651.688103] Lustre: 26770:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 651.698645] Lustre: 26770:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 651.718908] Lustre: 26770:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 651.727366] Lustre: 26770:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 654.929938] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 654.939884] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 654.960215] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 654.976302] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 654.984492] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 654.990663] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 654.996457] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 655.003755] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 655.009174] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 655.012661] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 655.017539] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 655.022400] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.663753] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 659.676993] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 659.695410] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 659.703226] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.719937] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 659.738111] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.759738] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 659.767501] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.782723] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 659.797545] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.812131] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 659.825525] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 670.718105] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 670.740925] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 670.750962] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 670.759896] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 670.772112] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 670.783428] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 670.796741] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 670.804623] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 670.812959] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 670.825587] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 670.838611] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 670.850609] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 721.889126] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 721.893760] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 721.912792] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 724.311288] Lustre: server umount lustre-OST07c6 complete [ 733.294853] Lustre: server umount lustre-MDT0000 complete [ 747.913884] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 750.700399] Key type lgssc unregistered [ 750.958736] LNet: 28728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 750.962856] LNetError: 28728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 750.976702] LNet: Removed LNI 192.168.201.137@tcp [ 751.787468] Key type .llcrypt unregistered [ 751.790615] Key type ._llcrypt unregistered [ 771.862562] Key type ._llcrypt registered [ 771.865125] Key type .llcrypt registered [ 772.026483] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 773.340726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 773.531647] alg: No test for adler32 (adler32-zlib) [ 774.752385] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 775.063321] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 776.760166] Key type lgssc registered [ 777.824459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 786.050275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 786.061378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 787.506053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 791.597449] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 797.670624] LustreError: 29999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 797.700735] LustreError: 29999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 799.626312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 800.013979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 804.363385] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 811.061301] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 816.268437] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 822.636386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 823.018023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 828.720303] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 832.242801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 835.363895] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 849.911991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 849.915153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 849.951304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 852.974733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 852.984637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 852.990405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 855.591320] Lustre: server umount lustre-OST0000 complete [ 861.664815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 861.677704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 861.689424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 861.692628] Lustre: Skipped 1 previous similar message [ 864.830031] Lustre: server umount lustre-MDT0000 complete [ 868.195127] LustreError: 29979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787734454 with bad export cookie 12771387705849034905 [ 868.208681] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 868.211657] LustreError: 29979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 868.757764] Lustre: server umount lustre-MDT0001 complete [ 879.914381] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 882.711357] Key type lgssc unregistered [ 883.090155] LNet: 33737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.102957] LNetError: 33737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.123697] LNet: Removed LNI 192.168.201.137@tcp [ 883.966284] Key type .llcrypt unregistered [ 883.968509] Key type ._llcrypt unregistered [ 899.616729] Key type ._llcrypt registered [ 899.617989] Key type .llcrypt registered [ 899.719892] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 900.765697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 900.781711] alg: No test for adler32 (adler32-zlib) [ 901.892289] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 902.135690] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 903.831169] Key type lgssc registered [ 905.080419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 911.505805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 911.522663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 913.041528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 917.050329] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 922.594618] LustreError: 34988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 922.615111] LustreError: 34988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 924.422294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 924.808333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 929.279703] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 936.421338] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 941.358545] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 947.355599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 947.685693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 953.481332] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 957.952115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 960.391211] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 970.219649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 970.229581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 970.261201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 973.283111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 973.284624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 975.751786] Lustre: server umount lustre-OST0000 complete [ 980.966492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 980.974168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 980.993436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 981.002117] Lustre: Skipped 1 previous similar message [ 983.520977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 983.522124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 983.553909] Lustre: Skipped 1 previous similar message [ 985.572389] Lustre: server umount lustre-MDT0000 complete [ 988.641277] LustreError: 36828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 988.656896] LustreError: 36828:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 988.734719] LustreError: 34969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787734574 with bad export cookie 10490328202400553846 [ 988.749831] LustreError: 34969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 988.755217] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 989.095704] Lustre: server umount lustre-MDT0001 complete [ 999.558634] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1001.980576] Key type lgssc unregistered [ 1002.203763] LNet: 38522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.207899] LNetError: 38522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.241791] LNet: Removed LNI 192.168.201.137@tcp [ 1003.995296] Key type .llcrypt unregistered [ 1003.997733] Key type ._llcrypt unregistered [ 1026.131451] Key type ._llcrypt registered [ 1026.132992] Key type .llcrypt registered [ 1026.219394] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1027.184514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1027.212100] alg: No test for adler32 (adler32-zlib) [ 1028.367424] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 1028.545531] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1030.215353] Key type lgssc registered [ 1031.532027] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1039.681699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1039.703616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1041.287747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1045.809972] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1051.621839] LustreError: 39941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1051.641016] LustreError: 39941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1054.727440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1055.180334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1060.630689] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1067.585330] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1072.720818] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1079.369148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1079.876350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1086.258298] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1087.038930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1093.350547] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1101.284800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1101.288714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1101.289768] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 1101.307821] Lustre: Skipped 1 previous similar message [ 1102.104360] Lustre: server umount lustre-MDT0000 complete [ 1102.308202] LustreError: 39942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1102.320288] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1102.339216] Lustre: Skipped 1 previous similar message [ 1105.139218] LustreError: 39922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787734691 with bad export cookie 9187278154346091007 [ 1105.143379] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1105.147634] LustreError: 39922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1106.385387] LustreError: 41781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1106.406543] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [ 1106.415426] LustreError: 41781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1107.428976] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1107.443630] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1110.432614] LustreError: 39936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1110.432981] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1110.449416] LustreError: 39936:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1111.542525] Lustre: server umount lustre-MDT0001 complete [ 1132.496381] Lustre: server umount lustre-OST0000 complete [ 1148.381273] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1151.142480] Key type lgssc unregistered [ 1151.438728] LNet: 43808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1151.450741] LNetError: 43808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1151.472397] LNet: Removed LNI 192.168.201.137@tcp [ 1152.254366] Key type .llcrypt unregistered [ 1152.259235] Key type ._llcrypt unregistered [ 1174.768365] Key type ._llcrypt registered [ 1174.769922] Key type .llcrypt registered [ 1174.831746] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1175.751166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1175.799350] alg: No test for adler32 (adler32-zlib) [ 1176.890662] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 1177.143129] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1178.823169] Key type lgssc registered [ 1179.749332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1187.859508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1187.871037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1189.251982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1193.086871] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1199.591689] LustreError: 45224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1199.605911] LustreError: 45224:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1200.781673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1201.454646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1205.393164] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1212.006076] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1217.519626] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1223.419718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1223.718446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1229.111498] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1234.989776] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1236.522861] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1244.643253] Lustre: lustre-OST0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 1245.153199] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1245.156511] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1245.165948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1245.175735] Lustre: Skipped 1 previous similar message [ 1245.196290] Lustre: Skipped 1 previous similar message [ 1249.643228] Lustre: server umount lustre-OST0000 complete [ 1268.192907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1268.203699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.219427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1270.754665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1270.753607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1270.772847] Lustre: Skipped 1 previous similar message [ 1274.692795] Lustre: server umount lustre-MDT0000 complete [ 1275.879559] LustreError: 46292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1275.915490] LustreError: 46292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1278.157987] LustreError: 45206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787734864 with bad export cookie 2770507145789961883 [ 1278.160757] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1278.166776] LustreError: 45206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1284.619894] Lustre: server umount lustre-MDT0001 complete [ 1294.815462] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1297.582236] Key type lgssc unregistered [ 1297.916646] LNet: 48948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1297.935614] LNetError: 48948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1298.986491] LNet: Removed LNI 192.168.201.137@tcp [ 1299.788870] Key type .llcrypt unregistered [ 1299.794554] Key type ._llcrypt unregistered [ 1322.905785] Key type ._llcrypt registered [ 1322.908714] Key type .llcrypt registered [ 1322.993406] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1324.077824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1324.148746] alg: No test for adler32 (adler32-zlib) [ 1325.291119] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 1325.521655] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1327.151428] Key type lgssc registered [ 1328.710790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1335.164890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1335.178127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1336.572097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1340.625458] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1347.049465] LustreError: 50381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1347.072258] LustreError: 50381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1348.483372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.794322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1353.044766] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1359.414486] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1364.458386] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1369.244689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1369.496044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1373.953289] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1379.996173] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1382.842726] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1388.967839] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1391.559521] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1403.339253] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-26 5h03m08s [ 1404.429806] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1406.445359] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 1406.446082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1406.466025] Lustre: Skipped 2 previous similar messages [ 1407.503927] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 1407.516709] Lustre: Skipped 4 previous similar messages [ 1412.321778] Lustre: server umount lustre-MDT0000 complete [ 1413.583231] LustreError: 50374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1414.988844] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1418.642993] LustreError: 50374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1418.681851] LustreError: 50374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1419.549392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1419.650848] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1419.852118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1419.888470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1420.912333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1422.370324] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1424.529488] Lustre: DEBUG MARKER: Wait for df (36707) ... [ 1425.692500] Lustre: DEBUG MARKER: done [ 1429.984070] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1429.987667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1429.988852] LustreError: Skipped 1 previous similar message [ 1429.992848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1430.001635] Lustre: Skipped 1 previous similar message [ 1430.026677] Lustre: Skipped 4 previous similar messages [ 1434.944919] Lustre: server umount lustre-OST0000 complete [ 1441.761425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1441.768326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1441.789858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1441.805720] Lustre: Skipped 1 previous similar message [ 1441.828445] Lustre: Skipped 3 previous similar messages [ 1444.195102] Lustre: server umount lustre-MDT0000 complete [ 1447.552600] LustreError: 51115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787735033 with bad export cookie 11127934720999838534 [ 1447.555630] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1447.567239] LustreError: 51115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1447.835901] Lustre: server umount lustre-MDT0001 complete [ 1458.399771] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1461.355529] Key type lgssc unregistered [ 1461.577654] LNet: 55307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1461.581392] LNetError: 55307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1461.599377] LNet: Removed LNI 192.168.201.137@tcp [ 1462.481248] Key type .llcrypt unregistered [ 1462.483285] Key type ._llcrypt unregistered [ 1498.117865] Key type ._llcrypt registered [ 1498.119701] Key type .llcrypt registered [ 1498.216611] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1499.491369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1499.516360] alg: No test for adler32 (adler32-zlib) [ 1500.595809] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 1500.849258] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1502.543637] Key type lgssc registered [ 1503.732670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1510.917514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1510.931950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1512.202456] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1512.241315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1512.438484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1516.700110] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1524.188578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1524.278930] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1524.302697] Lustre: 57512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1524.472228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1528.507378] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1535.040575] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1540.158711] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1546.135301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1546.257322] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1546.385382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1551.395289] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1554.078311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1557.082808] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1562.571962] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-26 5h05m47s [ 1564.083426] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1571.579481] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1573.597377] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1573.773212] LustreError: 58640:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1593.854681] Lustre: lustre-MDT0000: Client f0d6b4ee-a7df-4db5-b956-bb72dbb8932c (at 192.168.201.37@tcp) reconnecting [ 1613.791125] LustreError: 58640:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1615.970262] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1618.257244] Lustre: DEBUG MARKER: done [ 1625.568487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1625.572076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1625.579408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1626.080073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1626.085092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1626.090732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1626.097462] Lustre: Skipped 1 previous similar message [ 1629.333323] Lustre: server umount lustre-OST0000 complete [ 1636.325170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1636.331559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1636.337091] Lustre: Skipped 1 previous similar message [ 1636.357143] Lustre: Skipped 1 previous similar message [ 1643.492215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1643.497815] Lustre: Skipped 3 previous similar messages [ 1647.071118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1647.264814] Lustre: server umount lustre-MDT0000 complete [ 1650.544994] LustreError: 57506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787735236 with bad export cookie 6370369344775142064 [ 1650.550264] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1650.562926] LustreError: 57506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1650.910876] Lustre: server umount lustre-MDT0001 complete [ 1660.747300] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1663.331297] Key type lgssc unregistered [ 1663.552892] LNet: 61842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1663.566581] LNetError: 61842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1663.583659] LNet: Removed LNI 192.168.201.137@tcp [ 1664.293261] Key type .llcrypt unregistered [ 1664.296826] Key type ._llcrypt unregistered [ 1698.607613] Key type ._llcrypt registered [ 1698.609113] Key type .llcrypt registered [ 1698.721508] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1699.956166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1699.991820] alg: No test for adler32 (adler32-zlib) [ 1700.991846] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 1701.265848] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1702.943121] Key type lgssc registered [ 1703.979863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1707.130756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1707.145713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1708.347276] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1708.375253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1708.525701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1709.301585] Lustre: Failing over lustre-MDT0000 [ 1709.566059] Lustre: server umount lustre-MDT0000 complete [ 1715.829188] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 05:08:21 (1787735301) [ 1721.122104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1721.420419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1724.752298] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1731.626047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1731.732175] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1731.754053] Lustre: 64617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1731.998337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1736.151663] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1742.212985] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1747.044819] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1752.682982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1752.885166] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1753.099839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1758.486695] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1760.370841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1765.422672] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1771.093492] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1776.098922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1776.099889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1776.108680] Lustre: Skipped 2 previous similar messages [ 1776.111997] Lustre: Skipped 1 previous similar message [ 1777.519426] Lustre: server umount lustre-MDT0000 complete [ 1780.408295] LustreError: 64611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787735366 with bad export cookie 6425724400331112751 [ 1780.414431] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1780.416833] LustreError: 64611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1780.706434] Lustre: server umount lustre-MDT0001 complete [ 1783.183892] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1786.021319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1791.899230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1797.535143] Lustre: 63543:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787735367/real 1787735367] req@00000000fec6011f x1874576316337792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787735383 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1797.558258] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1801.631319] Lustre: 63545:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787735371/real 1787735371] req@000000005a3e833e x1874576316337920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787735387 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1806.029212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1806.069259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1807.007179] Lustre: 63543:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787735376/real 1787735376] req@0000000013d72bf7 x1874576316338304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787735392 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1807.033232] Lustre: 63543:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1809.058027] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1811.169209] Lustre: 63544:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787735381/real 1787735381] req@00000000d30b7c83 x1874576316338432/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787735397 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1812.453986] LustreError: 68385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1812.469079] LustreError: 68385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 1815.393511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1815.733803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1816.750663] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1819.254954] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1824.965756] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1829.413088] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1836.454268] Lustre: server umount lustre-MDT0000 complete [ 1836.518142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1836.524987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1836.542358] LustreError: 66467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1837.540766] LustreError: 69872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1837.556032] LustreError: 69872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1839.469663] LustreError: 64610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787735425 with bad export cookie 6425724400331115026 [ 1839.476496] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1839.479045] LustreError: 64610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1839.697237] Lustre: server umount lustre-MDT0001 complete [ 1844.755766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1851.651872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1859.039109] Lustre: 63543:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787735428/real 1787735428] req@00000000943cba7c x1874576316363392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787735444 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1859.057791] Lustre: 63543:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1859.064900] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1859.077926] Lustre: Skipped 2 previous similar messages [ 1864.505569] LustreError: 69118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1864.580161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1864.630537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1868.813508] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1869.746904] LustreError: 71065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1869.769019] LustreError: 71065:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1870.752087] Lustre: 63544:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787735440/real 1787735440] req@00000000b13e0394 x1874576316364032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787735456 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1870.768423] Lustre: 63544:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1875.835460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.226935] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1877.235019] Lustre: Skipped 1 previous similar message [ 1880.712924] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1887.412905] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1892.099849] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1897.952413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1897.963508] Lustre: Skipped 2 previous similar messages [ 1897.963513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1901.540334] Lustre: server umount lustre-MDT0000 complete [ 1903.078160] LustreError: 71062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1903.095698] LustreError: 71062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1904.442038] LustreError: 65355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787735490 with bad export cookie 6425724400331116419 [ 1904.447808] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1904.456589] LustreError: 65355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1904.689635] Lustre: server umount lustre-MDT0001 complete [ 1909.753871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1912.447200] Lustre: DEBUG MARKER: files compared the same [ 1924.457935] Lustre: server umount lustre-OST0000 complete [ 1937.331579] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1939.322488] Key type lgssc unregistered [ 1939.567543] LNet: 74341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1939.570758] LNetError: 74341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1939.582111] LNet: Removed LNI 192.168.201.137@tcp [ 1940.334957] Key type .llcrypt unregistered [ 1940.336789] Key type ._llcrypt unregistered [ 1959.612468] Key type ._llcrypt registered [ 1959.615479] Key type .llcrypt registered [ 1959.710638] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1960.692806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1960.813515] alg: No test for adler32 (adler32-zlib) [ 1961.836563] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 1961.999741] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1963.631150] Key type lgssc registered [ 1964.623787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1970.977539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1970.989677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1972.363058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1975.886891] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug +malloc all [ 1982.538227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1982.762159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1985.563496] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug +malloc all [ 1990.894650] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1995.320379] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2000.360614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2000.692921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2005.822823] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug +malloc all [ 2011.827486] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2018.086839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2028.513197] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2028.523844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2028.540246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2033.635628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2033.643547] Lustre: Skipped 2 previous similar messages [ 2034.247879] Lustre: server umount lustre-OST0000 complete [ 2038.756390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2038.779449] Lustre: Skipped 1 previous similar message [ 2038.785491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2043.871894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2043.879135] Lustre: Skipped 2 previous similar messages [ 2044.207081] Lustre: server umount lustre-MDT0000 complete [ 2047.011912] LustreError: 75717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787735632 with bad export cookie 11919591638207427495 [ 2047.025531] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2047.296656] Lustre: server umount lustre-MDT0001 complete [ 2056.003937] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2058.319511] Key type lgssc unregistered [ 2058.565196] LNet: 79294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2058.572311] LNetError: 79294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2058.588687] LNet: Removed LNI 192.168.201.137@tcp [ 2059.177581] Key type .llcrypt unregistered [ 2059.179531] Key type ._llcrypt unregistered [ 2076.906280] Key type ._llcrypt registered [ 2076.909021] Key type .llcrypt registered [ 2076.999241] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2077.598599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2077.680825] alg: No test for adler32 (adler32-zlib) [ 2078.766410] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 2078.998989] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2080.655163] Key type lgssc registered [ 2081.615513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2087.364546] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2087.385216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2102.879355] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2108.000095] LustreError: 80663:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.137@tcp: failed processing log, type 4: rc = -110 [ 2134.692916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2139.702923] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2146.069128] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2152.329027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.485351] Lustre: *** cfs_fail_loc=706, val=0*** [ 2152.802795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2152.891343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2156.610445] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2158.053658] LustreError: 80687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2158.076596] LustreError: 80687:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2163.174315] LustreError: 80688:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2163.333436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2163.535516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2167.562130] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2173.752510] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2178.024061] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2180.066488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2180.070043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2180.084073] Lustre: Skipped 1 previous similar message [ 2185.187818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2185.197510] Lustre: Skipped 2 previous similar messages [ 2185.797475] Lustre: server umount lustre-OST0000 complete [ 2189.280382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2189.291890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.307120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2190.310629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2195.418358] Lustre: server umount lustre-MDT0000 complete [ 2195.434213] LustreError: 81752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2195.461512] LustreError: 81752:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2198.681691] LustreError: 80671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787735784 with bad export cookie 8683775300422036594 [ 2198.686520] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2198.700109] LustreError: 80671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2198.959086] Lustre: server umount lustre-MDT0001 complete [ 2207.777541] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2210.900536] Key type lgssc unregistered [ 2211.255850] LNet: 84276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2211.264717] LNetError: 84276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2211.275986] LNet: Removed LNI 192.168.201.137@tcp [ 2212.570848] Key type .llcrypt unregistered [ 2212.572380] Key type ._llcrypt unregistered [ 2237.699479] Key type ._llcrypt registered [ 2237.701961] Key type .llcrypt registered [ 2237.794922] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2238.375094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2238.446903] alg: No test for adler32 (adler32-zlib) [ 2239.371981] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 2239.562493] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2241.255175] Key type lgssc registered [ 2241.935573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2248.524517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2248.540070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2253.993459] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2260.513188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.838313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2264.947797] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2266.092083] LustreError: 86381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2266.124986] LustreError: 86381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2270.178050] LustreError: 86380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2270.189734] LustreError: 86380:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2272.574194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2272.952784] LustreError: 86381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2272.979707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2272.980741] LustreError: 86381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2278.279772] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2278.373696] LustreError: 86380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2278.393933] LustreError: 86380:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2283.508497] LustreError: 86379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2283.523108] LustreError: 86379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2286.610897] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2286.620736] Lustre: MGS: Client c358517b-6a5a-45aa-811c-6543faeddf0e (at 0@lo) reconnecting [ 2286.630909] LustreError: 86395:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2286.883081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2286.962922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2291.461665] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2307.559356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2307.566646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2307.570410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2307.587112] Lustre: Skipped 1 previous similar message [ 2311.485819] Lustre: server umount lustre-OST0000 complete [ 2317.795439] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2317.797077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2317.802618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2317.802626] Lustre: Skipped 1 previous similar message [ 2317.810207] Lustre: Skipped 1 previous similar message [ 2320.622825] Lustre: server umount lustre-MDT0000 complete [ 2322.925868] LustreError: 88297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2322.953296] LustreError: 88297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2324.695960] Lustre: server umount lustre-MDT0001 complete [ 2328.425425] Lustre: server umount MGS complete [ 2337.593805] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 05:18:42 (1787735922) [ 2362.068538] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 2370.812604] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2405.933743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2410.493860] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2418.114399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2418.266618] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2418.306399] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2418.358568] Lustre: lustre-MDT0001: new disk, initializing [ 2418.415944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2418.434835] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2418.442951] Lustre: 93559:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2418.447179] LustreError: 93559:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2422.523735] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2431.082813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2431.312273] Lustre: lustre-OST0000: new disk, initializing [ 2431.315111] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2431.318258] Lustre: 94325:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2431.378851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2437.119699] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2441.766966] Lustre: 94812:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004d94c426 x1874576880475392/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787736043 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2446.567169] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2452.545667] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2452.558618] Lustre: MGS: Client cce634f9-4071-4a52-b9cb-2b0844b9455e (at 0@lo) reconnecting [ 2452.625868] Lustre: 92829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2452.645939] LustreError: 93545:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2452.657112] LustreError: 93545:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2452.687455] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2452.811318] Lustre: lustre-MDT0000: new disk, initializing [ 2453.059922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2453.091534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2453.236804] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2454.179862] Lustre: 95394:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000dd9f4f4e x1874576880482304/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787736056 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2458.074484] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2462.184661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2462.197819] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2463.276394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2463.286396] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2463.420889] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2487.780358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2487.795309] Lustre: Skipped 1 previous similar message [ 2487.796984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2487.801676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2487.823119] Lustre: Skipped 2 previous similar messages [ 2492.897936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2492.907158] Lustre: Skipped 1 previous similar message [ 2493.477764] Lustre: server umount lustre-OST0000 complete [ 2498.016621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2498.023061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2498.027417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2498.037559] Lustre: Skipped 1 previous similar message [ 2498.045191] Lustre: Skipped 1 previous similar message [ 2503.992404] Lustre: server umount lustre-MDT0000 complete [ 2508.270973] LustreError: 93534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.288521] LustreError: 93534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2514.713682] Lustre: server umount lustre-MDT0001 complete [ 2518.421751] Lustre: server umount MGS complete [ 2529.984946] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 05:21:54 (1787736114) [ 2552.111851] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2554.920573] Key type lgssc unregistered [ 2555.225790] LNet: 97900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2555.233061] LNetError: 97900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2555.259524] LNet: Removed LNI 192.168.201.137@tcp [ 2556.527991] Key type .llcrypt unregistered [ 2556.531161] Key type ._llcrypt unregistered [ 2574.888674] Key type ._llcrypt registered [ 2574.891995] Key type .llcrypt registered [ 2575.000266] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2576.075951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2576.217934] alg: No test for adler32 (adler32-zlib) [ 2577.506163] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 2577.802961] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2579.480314] Key type lgssc registered [ 2581.173824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2589.015887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2589.031067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2590.570707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2593.272295] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2600.431402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2600.725731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2606.004731] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2612.791696] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2617.958874] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2624.246421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2624.511799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2630.762724] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2636.789035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2637.291869] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2645.984981] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2645.993589] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2646.012236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2647.012834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2647.013814] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2647.013832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2647.045587] Lustre: Skipped 1 previous similar message [ 2651.561151] Lustre: server umount lustre-OST0000 complete [ 2657.248976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2657.249911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2657.251192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2657.274677] Lustre: Skipped 1 previous similar message [ 2660.470014] Lustre: server umount lustre-MDT0000 complete [ 2662.375718] LustreError: 99123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2662.395920] LustreError: 99123:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2663.767995] LustreError: 99102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787736249 with bad export cookie 12894432689646319250 [ 2663.774430] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2663.775821] LustreError: 99102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2664.040168] Lustre: server umount lustre-MDT0001 complete [ 2675.137236] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2678.001564] Key type lgssc unregistered [ 2678.336965] LNet: 102659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2678.346207] LNetError: 102659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2678.364299] LNet: Removed LNI 192.168.201.137@tcp [ 2679.340303] Key type .llcrypt unregistered [ 2679.342718] Key type ._llcrypt unregistered [ 2694.731447] Key type ._llcrypt registered [ 2694.734672] Key type .llcrypt registered [ 2694.851526] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2695.899589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2695.982094] alg: No test for adler32 (adler32-zlib) [ 2697.008313] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 2697.263649] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2698.943957] Key type lgssc registered [ 2699.885785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2704.532824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2704.542349] LustreError: 103723:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 2704.643302] LustreError: 103743:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 2704.647959] LustreError: 103723:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4904 [ 2704.654579] LustreError: 103723:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2704.660835] LustreError: 103723:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2704.663191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2706.109528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2712.925206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2713.136164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2716.785798] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2720.691087] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing lsmod [ 2722.900323] LustreError: 104935:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 2722.967573] LustreError: 104949:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 2722.971738] LustreError: 104935:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4932 [ 2722.975339] LustreError: 104935:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2722.979069] LustreError: 104935:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2722.981273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2723.316821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2728.418517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2728.426428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.453634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2729.449645] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2729.449904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2729.464408] Lustre: Skipped 1 previous similar message [ 2729.479037] Lustre: Skipped 2 previous similar messages [ 2731.796205] Lustre: server umount lustre-MDT0000 complete [ 2734.561077] LustreError: 103776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2734.581471] LustreError: 103776:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2735.145498] LustreError: 105137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787736321 with bad export cookie 14734696443119159017 [ 2735.160109] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2735.176530] Lustre: 103772:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2735.567243] Lustre: server umount lustre-MDT0001 complete [ 2750.167382] Lustre: server umount lustre-OST0000 complete [ 2758.480845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2758.950905] LustreError: 106015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2759.058653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2762.962372] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2764.257227] LustreError: 106014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2764.276513] LustreError: 106014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2768.356406] LustreError: 106015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2771.568143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2771.809562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2774.949671] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2780.417768] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2784.933982] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2790.480260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2790.761946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2796.085814] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2802.519232] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2803.065390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2813.410429] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2813.427386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2813.432681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2813.444291] Lustre: Skipped 1 previous similar message [ 2818.533343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2818.538596] Lustre: Skipped 2 previous similar messages [ 2818.814859] Lustre: server umount lustre-OST0000 complete [ 2823.142109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2823.162656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2823.185193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2827.718686] Lustre: server umount lustre-MDT0000 complete [ 2828.781446] LustreError: 106009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2828.813154] LustreError: 106009:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2830.812360] LustreError: 105995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787736416 with bad export cookie 14734696443119159689 [ 2830.817536] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2830.822392] LustreError: 105995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2831.183281] Lustre: server umount lustre-MDT0001 complete [ 2840.226542] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2842.862529] Key type lgssc unregistered [ 2843.238590] LNet: 109550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2843.243939] LNetError: 109550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2844.263760] LNet: Removed LNI 192.168.201.137@tcp [ 2845.036421] Key type .llcrypt unregistered [ 2845.037857] Key type ._llcrypt unregistered [ 2864.594651] Key type ._llcrypt registered [ 2864.595949] Key type .llcrypt registered [ 2864.660514] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2865.480822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2865.530448] alg: No test for adler32 (adler32-zlib) [ 2866.503492] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 2866.677383] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2868.303717] Key type lgssc registered [ 2869.215211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2876.146501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2876.174767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2877.623347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2881.642218] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2887.137814] LustreError: 110965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2887.152193] LustreError: 110965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2888.317869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2888.691733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2892.498945] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2898.921444] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2903.512763] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2908.964670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2909.364721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2914.373964] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2920.610930] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2922.696365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 2930.414907] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2933.176034] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2938.338195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2938.340120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2938.344934] Lustre: Skipped 1 previous similar message [ 2938.357721] Lustre: Skipped 1 previous similar message [ 2938.658121] Lustre: server umount lustre-OST0000 complete [ 2943.475117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2943.484811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2943.494826] Lustre: Skipped 1 previous similar message [ 2943.509335] Lustre: Skipped 1 previous similar message [ 2948.577015] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2948.581332] Lustre: Skipped 1 previous similar message [ 2953.697943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2953.706496] Lustre: Skipped 1 previous similar message [ 2957.279138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2957.505248] Lustre: server umount lustre-MDT0000 complete [ 2958.822293] LustreError: 110966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2958.844213] LustreError: 110966:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2961.013343] LustreError: 110946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787736546 with bad export cookie 18127624173043523620 [ 2961.014521] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2961.019040] LustreError: 110946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2961.276174] Lustre: server umount lustre-MDT0001 complete [ 2970.862430] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2973.361795] Key type lgssc unregistered [ 2973.694072] LNet: 114605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2973.700158] LNetError: 114605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2973.713176] LNet: Removed LNI 192.168.201.137@tcp [ 2974.410339] Key type .llcrypt unregistered [ 2974.412300] Key type ._llcrypt unregistered [ 2988.936926] Key type ._llcrypt registered [ 2988.939260] Key type .llcrypt registered [ 2989.013568] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2989.851431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2990.040036] alg: No test for adler32 (adler32-zlib) [ 2991.019419] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 2991.261643] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2992.959929] Key type lgssc registered [ 2993.894777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2999.989992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3000.010356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3001.354843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3005.230266] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3011.557311] LustreError: 115808:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3011.576312] LustreError: 115808:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3012.264035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3012.555071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3016.601573] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3022.511651] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3027.262209] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3032.859113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3033.161671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3038.534586] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3045.087656] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3046.474317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3056.613968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3056.616151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3056.616917] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3056.636939] Lustre: Skipped 1 previous similar message [ 3056.670633] Lustre: Skipped 1 previous similar message [ 3061.732146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3061.734431] Lustre: Skipped 1 previous similar message [ 3062.429984] Lustre: server umount lustre-OST0000 complete [ 3066.851681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3066.872440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3071.781780] Lustre: server umount lustre-MDT0000 complete [ 3071.969914] LustreError: 116861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3071.980859] LustreError: 116861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3075.248031] LustreError: 115789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787736661 with bad export cookie 5512540972485507214 [ 3075.252814] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3075.258052] LustreError: 115789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3075.727572] Lustre: server umount lustre-MDT0001 complete [ 3089.355604] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 3092.258678] Key type lgssc unregistered [ 3092.524752] LNet: 119344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3092.534467] LNetError: 119344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3093.607710] LNet: Removed LNI 192.168.201.137@tcp [ 3094.281314] Key type .llcrypt unregistered [ 3094.286369] Key type ._llcrypt unregistered [ 3113.618993] Key type ._llcrypt registered [ 3113.621142] Key type .llcrypt registered [ 3113.740348] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3114.661428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3114.841105] alg: No test for adler32 (adler32-zlib) [ 3115.897669] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 3116.088326] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 3117.791652] Key type lgssc registered [ 3118.831938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3125.083402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3125.105145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3126.578143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3130.957209] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3136.998440] LustreError: 120740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3137.021077] LustreError: 120740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3138.172284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3138.447083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3142.775851] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3149.577651] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3153.811598] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3159.601962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3159.919843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3165.981700] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3171.199381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3173.125270] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3179.718738] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3182.615961] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3182.623641] Lustre: Skipped 2 previous similar messages [ 3186.280634] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3186.294826] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3186.302079] Lustre: Skipped 2 previous similar messages [ 3186.307963] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3192.869320] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3192.874523] Lustre: Skipped 1 previous similar message [ 3197.558795] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.37@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3197.568220] Lustre: Skipped 2 previous similar messages [ 3199.651091] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.37@tcp 192.168.0.[2,10]@tcp [ 3202.429090] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.37/24@tcp in log lustre-MDT0000 [ 3202.434329] Lustre: Skipped 2 previous similar messages [ 3209.758893] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.37/24@tcp [ 3209.762627] Lustre: Skipped 1 previous similar message [ 3215.328431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3215.336694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3215.345886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3215.851957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3215.852841] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3215.866810] LustreError: Skipped 2 previous similar messages [ 3215.882532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3216.016763] Lustre: server umount lustre-OST0000 complete [ 3220.449662] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3220.461953] LustreError: Skipped 3 previous similar messages [ 3220.464581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3220.489385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3222.504672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3222.505090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3225.705586] Lustre: server umount lustre-MDT0000 complete [ 3227.617640] LustreError: 122801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3227.638750] LustreError: 122801:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3228.739817] LustreError: 120720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787736814 with bad export cookie 17030778864377128473 [ 3228.755320] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3228.759386] LustreError: 120720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3229.065478] Lustre: server umount lustre-MDT0001 complete [ 3239.480782] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 3242.539299] Key type lgssc unregistered [ 3242.890301] LNet: 125735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3242.897269] LNetError: 125735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3242.923385] LNet: Removed LNI 192.168.201.137@tcp [ 3243.956865] Key type .llcrypt unregistered [ 3243.963932] Key type ._llcrypt unregistered [ 3271.926815] Key type ._llcrypt registered [ 3271.928831] Key type .llcrypt registered [ 3272.005194] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3272.902564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3272.938798] alg: No test for adler32 (adler32-zlib) [ 3273.984460] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 3274.252499] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 3275.920365] Key type lgssc registered [ 3277.218918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3285.318405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3285.343781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3286.957513] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3286.979885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3286.998066] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.37/24@tcp [ 3292.783880] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3296.225300] LustreError: 127297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3296.251087] LustreError: 127297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3301.358465] LustreError: 127298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3303.116296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3303.554135] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3303.590176] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.37/24@tcp [ 3303.699501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3308.515763] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3315.162983] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3320.802388] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3326.934360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3327.445833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3331.595537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3333.798367] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3340.489351] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3352.034133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3352.052402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3352.064143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3352.069174] Lustre: Skipped 1 previous similar message [ 3356.139823] Lustre: server umount lustre-OST0000 complete [ 3362.276835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3362.284272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3362.300331] Lustre: Skipped 2 previous similar messages [ 3362.310241] Lustre: Skipped 1 previous similar message [ 3367.392568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3367.402183] Lustre: Skipped 1 previous similar message [ 3367.552114] Lustre: server umount lustre-MDT0000 complete [ 3371.854566] LustreError: 127279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787736957 with bad export cookie 13158795177730685834 [ 3371.873586] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3372.310771] Lustre: server umount lustre-MDT0001 complete [ 3385.250332] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 3388.501105] Key type lgssc unregistered [ 3388.885056] LNet: 130888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3388.887810] LNetError: 130888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3388.916905] LNet: Removed LNI 192.168.201.137@tcp [ 3390.278584] Key type .llcrypt unregistered [ 3390.281160] Key type ._llcrypt unregistered [ 3421.802447] Key type ._llcrypt registered [ 3421.805740] Key type .llcrypt registered [ 3421.947422] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 3437.607982] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3438.933699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3438.960061] alg: No test for adler32 (adler32-zlib) [ 3440.157195] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 3440.632216] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 3442.351252] Key type lgssc registered [ 3444.328678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3482.287685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3482.322651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3483.590157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3483.618936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3483.686888] Lustre: lustre-MDT0000: new disk, initializing [ 3483.794092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3483.813777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3488.772339] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3499.612995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3499.735598] Lustre: 134289:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3499.787755] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3499.790867] Lustre: Skipped 1 previous similar message [ 3499.851994] Lustre: lustre-MDT0001: new disk, initializing [ 3499.933379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3499.984958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3499.999288] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3504.853975] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3515.108549] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3520.743662] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3527.412866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3527.643147] Lustre: lustre-OST0000: new disk, initializing [ 3527.645606] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3527.653427] Lustre: 136261:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3527.731446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3534.097172] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3535.970089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3535.991142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3536.161284] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3543.096518] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3547.719541] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3547.916425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3552.402471] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3552.628418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3556.833778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3556.837955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3556.840913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3556.841882] LustreError: Skipped 1 previous similar message [ 3556.865634] Lustre: Skipped 1 previous similar message [ 3560.734715] Lustre: server umount lustre-OST0000 complete [ 3566.562713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3566.577093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3566.594918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3566.601510] Lustre: Skipped 1 previous similar message [ 3570.764355] Lustre: server umount lustre-MDT0000 complete [ 3574.147566] LustreError: 134283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787737160 with bad export cookie 10427874395242377905 [ 3574.147804] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3574.163675] LustreError: 134283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3574.467439] Lustre: server umount lustre-MDT0001 complete [ 3582.509349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3583.096759] LustreError: 138479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3583.112448] LustreError: 138479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3583.162867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3587.315589] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3588.576379] LustreError: 138480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3593.699395] LustreError: 138479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3595.045845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3595.320227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3599.861285] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3606.967271] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3612.922670] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3619.874914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3620.141561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3626.091840] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3633.524303] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3638.765767] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3639.065835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3643.780138] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3643.978990] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3653.139718] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3653.301103] Lustre: lustre-OST0001: new disk, initializing [ 3653.304070] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3653.310662] Lustre: 141851:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3653.397292] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3659.437490] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3661.958315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3661.974635] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3662.106035] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3667.681958] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3668.007940] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3672.761907] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3672.987767] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3678.308686] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9578075f9800.ost_server_uuid 50 [ 3679.708220] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9578075f9800.ost_server_uuid in FULL state after 0 sec [ 3693.037068] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3693.040461] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3693.055084] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3693.065567] Lustre: Skipped 2 previous similar messages [ 3693.073995] Lustre: Skipped 3 previous similar messages [ 3694.288844] Lustre: server umount lustre-OST0001 complete [ 3698.148456] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3698.149124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3698.149932] LustreError: 142316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3698.161687] LustreError: Skipped 1 previous similar message [ 3698.186205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3698.190639] Lustre: Skipped 1 previous similar message [ 3703.273453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3703.277278] LustreError: 140298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3703.282805] Lustre: Skipped 1 previous similar message [ 3703.307257] LustreError: 140298:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3704.092289] Lustre: server umount lustre-OST0000 complete [ 3707.872499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3707.881093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3707.890084] Lustre: Skipped 1 previous similar message [ 3713.505153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3713.509782] Lustre: Skipped 4 previous similar messages [ 3713.867348] Lustre: server umount lustre-MDT0000 complete [ 3717.289923] LustreError: 138459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787737303 with bad export cookie 10427874395242378780 [ 3717.295121] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3717.297529] LustreError: 138459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3717.540949] Lustre: server umount lustre-MDT0001 complete [ 3734.751449] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 3737.457198] Key type lgssc unregistered [ 3737.708922] LNet: 144856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3737.716071] LNetError: 144856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3737.739903] LNet: Removed LNI 192.168.201.137@tcp [ 3738.538570] Key type .llcrypt unregistered [ 3738.540811] Key type ._llcrypt unregistered [ 3778.793502] Key type ._llcrypt registered [ 3778.798679] Key type .llcrypt registered [ 3778.923579] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 3795.366961] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3797.204525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3797.234204] alg: No test for adler32 (adler32-zlib) [ 3798.429469] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 3798.801365] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 3800.495159] Key type lgssc registered [ 3801.729352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3831.232563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3831.261851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3832.549314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3832.580225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3832.671956] Lustre: lustre-MDT0000: new disk, initializing [ 3832.730564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3832.744758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3836.808227] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3846.233855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3846.371541] Lustre: 149015:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3846.430403] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3846.432978] Lustre: Skipped 1 previous similar message [ 3846.515197] Lustre: lustre-MDT0001: new disk, initializing [ 3846.615629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3846.642926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3846.652951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3850.524830] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3859.443777] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3864.386658] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3870.071460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3870.322258] Lustre: lustre-OST0000: new disk, initializing [ 3870.325619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3870.330699] Lustre: 150978:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3870.421263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3875.913132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3875.924214] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3876.035836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3876.933898] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3885.862721] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3891.166328] Lustre: Failing over lustre-OST0000 [ 3891.337495] Lustre: server umount lustre-OST0000 complete [ 3907.039695] Lustre: 147237:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787737477/real 1787737477] req@00000000e0c43d36 x1874578515197952/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787737493 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3907.057504] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3907.091926] Lustre: 147237:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 3907.127815] Lustre: Skipped 1 previous similar message [ 3908.107829] Lustre: 147234:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787737478/real 1787737478] req@00000000be696755 x1874578515198208/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787737494 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3909.793667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3910.160247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3910.174750] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3912.203098] Lustre: 147234:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787737482/real 1787737482] req@000000005d1c6232 x1874578515199232/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787737498 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3912.216807] Lustre: 147234:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3915.216026] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3916.089764] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3917.287531] Lustre: 147235:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787737487/real 1787737487] req@00000000a397deda x1874578515200384/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787737503 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3917.333356] Lustre: 147235:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3917.343590] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 3917.344066] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3919.426817] Lustre: Failing over lustre-MDT0000 [ 3919.695243] Lustre: server umount lustre-MDT0000 complete [ 3922.403203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3922.424241] Lustre: Skipped 1 previous similar message [ 3925.457683] LustreError: 149022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3925.468668] LustreError: 149022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3927.520670] LustreError: 149026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3927.537538] LustreError: 149026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3930.578181] LustreError: 149022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3932.641733] LustreError: 149026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3932.661956] LustreError: 149026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3937.761707] LustreError: 149025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3937.771424] LustreError: 149025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3938.403282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3938.522879] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3938.749717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3938.792362] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3940.825847] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3943.927153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3943.948213] Lustre: Skipped 1 previous similar message [ 3943.979713] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 3944.047093] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3944.049262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3953.123726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3953.127293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3953.160453] Lustre: Skipped 1 previous similar message [ 3953.175343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3954.145202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3958.324624] Lustre: server umount lustre-OST0000 complete [ 3964.386367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3964.392935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3964.417866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3964.436058] Lustre: Skipped 2 previous similar messages [ 3964.458718] Lustre: Skipped 2 previous similar messages [ 3968.507673] Lustre: server umount lustre-MDT0000 complete [ 3969.506821] LustreError: 149020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3969.527616] LustreError: 149020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3972.015817] LustreError: 149008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787737557 with bad export cookie 425391553324677098 [ 3972.021186] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3972.027166] LustreError: 149008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3972.365150] Lustre: server umount lustre-MDT0001 complete [ 3983.617921] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 3987.374963] Key type lgssc unregistered [ 3987.805645] LNet: 154652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3987.814331] LNetError: 154652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3987.836134] LNet: Removed LNI 192.168.201.137@tcp [ 3988.678236] Key type .llcrypt unregistered [ 3988.679734] Key type ._llcrypt unregistered [ 4023.810199] Key type ._llcrypt registered [ 4023.811637] Key type .llcrypt registered [ 4023.897862] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4024.717601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4024.868688] alg: No test for adler32 (adler32-zlib) [ 4025.897237] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 4026.104473] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4027.743205] Key type lgssc registered [ 4028.807958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4035.734700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4035.753797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4037.072724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4041.096586] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4047.334185] LustreError: 156171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4047.355452] LustreError: 156171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4050.945744] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4052.472498] LustreError: 156172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4054.872397] LustreError: 156200:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4054.880717] LustreError: 156200:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4054.884897] LustreError: 156200:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4055.178312] Lustre: server umount lustre-MDT0000 complete [ 4071.447038] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 4081.559694] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4117.637324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4117.887331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4117.910254] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4117.976815] Lustre: lustre-MDT0000: new disk, initializing [ 4118.065910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4118.080367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4122.916210] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4134.720859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4134.864654] Lustre: 159502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4134.926831] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4134.932831] Lustre: Skipped 1 previous similar message [ 4135.000752] Lustre: lustre-MDT0001: new disk, initializing [ 4135.092947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4135.150288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4135.165701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4139.784861] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4150.721780] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4157.324859] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4164.202896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4164.668195] Lustre: lustre-OST0000: new disk, initializing [ 4164.675362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4164.689322] Lustre: 161470:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4164.802202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4170.337230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4170.353169] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4170.506473] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4171.153066] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4181.046902] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4185.547978] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4186.975912] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 05:49:32 (1787737772) [ 4411.362526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4411.364437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4411.388319] Lustre: Skipped 2 previous similar messages [ 4411.391362] Lustre: Skipped 1 previous similar message [ 4414.177574] Lustre: server umount lustre-MDT0000 complete [ 4416.480478] LustreError: 163240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4416.536285] LustreError: 163240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4419.154721] LustreError: 161481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787738005 with bad export cookie 13246226041688146334 [ 4419.158941] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4419.170021] LustreError: 161481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4421.601142] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4421.601610] LustreError: 159510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4421.618897] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4421.618906] Lustre: Skipped 1 previous similar message [ 4424.610294] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4425.580944] Lustre: server umount lustre-MDT0001 complete [ 4436.338300] Lustre: server umount lustre-OST0000 complete [ 4446.562602] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 4457.215725] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4494.132980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4494.451783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4494.500570] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4494.617334] Lustre: lustre-MDT0000: new disk, initializing [ 4494.710098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4494.729431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4499.782890] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4510.446395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4510.598233] Lustre: 166125:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4510.668721] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4510.676281] Lustre: Skipped 1 previous similar message [ 4510.779263] Lustre: lustre-MDT0001: new disk, initializing [ 4510.913238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4510.976198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4510.995099] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4515.893992] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4525.111201] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4530.553049] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4538.092375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4538.369114] Lustre: lustre-OST0000: new disk, initializing [ 4538.378309] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4538.385921] Lustre: 168099:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4538.492168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4545.105298] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4545.618253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4545.641705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4545.784271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4553.668177] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4557.378410] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4557.617516] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4561.716659] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4561.980853] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4566.499518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4566.503911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4566.532786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4571.618803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4571.621482] Lustre: Skipped 1 previous similar message [ 4578.271234] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4578.429739] Lustre: server umount lustre-OST0000 complete [ 4582.880749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4582.894161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.909642] Lustre: Skipped 1 previous similar message [ 4582.926582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4582.933442] Lustre: Skipped 2 previous similar messages [ 4586.984039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4589.029840] Lustre: server umount lustre-MDT0000 complete [ 4592.096893] LustreError: 166139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4592.134199] LustreError: 166139:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4592.972553] LustreError: 166931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787738178 with bad export cookie 13246226041688305164 [ 4592.976071] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4592.994158] LustreError: 166931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4593.373050] Lustre: server umount lustre-MDT0001 complete [ 4603.001416] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:56:27 (1787738187) [ 4617.962938] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 4626.926600] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4662.024021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4662.281231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4662.318242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4662.407342] Lustre: lustre-MDT0000: new disk, initializing [ 4662.489557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4662.505684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4667.712623] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4678.645749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4678.834187] Lustre: 172856:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4679.193077] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4684.174235] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4693.896806] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4699.571663] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4705.967751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4706.219530] Lustre: lustre-OST0000: new disk, initializing [ 4706.222861] Lustre: Skipped 1 previous similar message [ 4706.226570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4706.230514] Lustre: Skipped 2 previous similar messages [ 4706.234357] Lustre: 174831:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4712.501072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4712.508726] Lustre: Skipped 1 previous similar message [ 4712.535167] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4712.637374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4712.727297] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4720.851920] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4728.288657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4728.290143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4728.294484] LustreError: Skipped 1 previous similar message [ 4728.295071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4728.309376] Lustre: Skipped 1 previous similar message [ 4728.313059] Lustre: Skipped 2 previous similar messages [ 4728.881282] Lustre: server umount lustre-OST0000 complete [ 4738.738752] Lustre: server umount lustre-MDT0000 complete [ 4742.508224] LustreError: 172847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787738328 with bad export cookie 13246226041688306046 [ 4742.536216] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4742.906435] Lustre: server umount lustre-MDT0001 complete [ 4752.005617] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:58:56 (1787738336) [ 4771.195990] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 4779.892797] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4815.992182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4816.191613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4816.216033] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4816.305597] Lustre: lustre-MDT0000: new disk, initializing [ 4816.395947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4816.406232] Lustre: Skipped 2 previous similar messages [ 4816.415643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4822.057604] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4832.200348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4832.380842] Lustre: 179151:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4832.803995] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4838.361451] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4848.703230] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4853.710961] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4860.143050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4860.435453] Lustre: 181132:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4867.029773] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4868.263853] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4868.447566] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4877.309412] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4889.060934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4889.061518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4889.073292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4889.073301] Lustre: Skipped 5 previous similar messages [ 4889.089762] Lustre: Skipped 3 previous similar messages [ 4889.130853] LustreError: Skipped 1 previous similar message [ 4890.803592] Lustre: server umount lustre-OST0000 complete [ 4899.296879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4903.287167] Lustre: server umount lustre-MDT0000 complete [ 4904.425930] LustreError: 179165:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4904.449769] LustreError: 179165:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4907.075452] LustreError: 179145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787738493 with bad export cookie 13246226041688307131 [ 4907.076662] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4907.101954] LustreError: 179145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4907.552698] Lustre: server umount lustre-MDT0001 complete [ 4919.842767] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 4922.583136] Key type lgssc unregistered [ 4922.863438] LNet: 183002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4922.872133] LNetError: 183002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4923.941960] LNet: Removed LNI 192.168.201.137@tcp [ 4924.907678] Key type .llcrypt unregistered [ 4924.909867] Key type ._llcrypt unregistered [ 4950.008501] Key type ._llcrypt registered [ 4950.010415] Key type .llcrypt registered [ 4950.140436] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4951.746079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4951.945238] alg: No test for adler32 (adler32-zlib) [ 4953.222945] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 4953.539966] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4955.191368] Key type lgssc registered [ 4956.461812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4964.419513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4964.437265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4965.955196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4970.716105] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4975.076063] LustreError: 184419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4975.095308] LustreError: 184419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4979.326057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4979.700828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4984.543683] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4991.679425] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4997.138607] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5002.703042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5003.080036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5008.695869] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5010.224586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5015.310451] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5034.980499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5034.985745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5035.011382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5036.000471] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5036.008639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5036.010812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5036.044986] Lustre: Skipped 1 previous similar message [ 5039.707567] Lustre: server umount lustre-OST0000 complete [ 5046.261343] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5046.265339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5046.281072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5046.299484] Lustre: Skipped 1 previous similar message [ 5049.204360] Lustre: server umount lustre-MDT0000 complete [ 5051.361020] LustreError: 184415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5051.373941] LustreError: 184415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5052.898320] LustreError: 184399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787738638 with bad export cookie 1683683686269463574 [ 5052.911323] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5052.911990] LustreError: 184399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5053.155745] Lustre: server umount lustre-MDT0001 complete [ 5064.779725] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 5067.808824] Key type lgssc unregistered [ 5068.149901] LNet: 187978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5068.156419] LNetError: 187978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5068.170374] LNet: Removed LNI 192.168.201.137@tcp [ 5068.978537] Key type .llcrypt unregistered [ 5068.981213] Key type ._llcrypt unregistered [ 5090.809968] Key type ._llcrypt registered [ 5090.812588] Key type .llcrypt registered [ 5090.959683] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5092.102266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5092.388347] alg: No test for adler32 (adler32-zlib) [ 5093.493030] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 5093.855803] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 5095.583974] Key type lgssc registered [ 5096.977681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5104.790450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5104.820079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5106.178636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5110.289585] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5116.404064] LustreError: 189394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5116.416186] LustreError: 189394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5117.884383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5118.332069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5122.520599] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5128.653613] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5134.046246] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5140.723651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5141.217990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5148.107986] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5151.504772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5155.570249] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5161.955331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5161.968616] Lustre: Skipped 1 previous similar message [ 5161.976354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5163.520312] Lustre: lustre-OST0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 5163.526420] Lustre: Skipped 1 previous similar message [ 5167.075517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5167.806200] Lustre: server umount lustre-OST0000 complete [ 5175.847796] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9578050ea000.ost_server_uuid 50 [ 5177.523581] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5192.612704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5192.618724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5192.632402] Lustre: Skipped 1 previous similar message [ 5192.639904] Lustre: Skipped 2 previous similar messages [ 5197.794435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5197.797298] Lustre: Skipped 1 previous similar message [ 5197.880994] Lustre: server umount lustre-MDT0000 complete [ 5202.103374] LustreError: 189376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787738788 with bad export cookie 3049616563210804367 [ 5202.111356] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5202.117846] LustreError: 189376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5202.561738] Lustre: server umount lustre-MDT0001 complete [ 5210.785126] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 06:06:35 (1787738795) [ 5217.294937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5217.823408] LustreError: 193311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5217.918809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5221.659903] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5223.395381] LustreError: 193312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5227.490113] LustreError: 193311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5229.526300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5229.882026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5234.386228] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5241.286370] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5246.245162] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5252.550386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5252.843275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5258.683773] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5262.083205] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5265.606561] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5272.106538] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5272.280478] Lustre: lustre-OST0001: new disk, initializing [ 5272.286959] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5272.292510] Lustre: 196149:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5272.384788] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5278.646831] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5280.292777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5280.302184] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5280.375785] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5287.750666] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5290.479553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5290.485299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5290.494924] Lustre: Skipped 1 previous similar message [ 5304.287256] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5304.379206] Lustre: server umount lustre-OST0000 complete [ 5305.831693] LustreError: 196579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5305.855537] LustreError: 196579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5309.965612] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5310.186591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5314.002121] LustreError: 196579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5314.020909] LustreError: 196579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5314.837594] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5315.035330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5331.426242] LustreError: 195133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5331.427886] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5331.430580] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5331.432894] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5331.432902] Lustre: Skipped 8 previous similar messages [ 5331.450611] LustreError: 195133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 5334.447378] Lustre: server umount lustre-OST0001 complete [ 5337.568282] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5337.574951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5337.585014] Lustre: Skipped 1 previous similar message [ 5343.704540] Lustre: server umount lustre-MDT0000 complete [ 5346.700795] LustreError: 193291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787738932 with bad export cookie 3049616563210805438 [ 5346.704080] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5346.711397] LustreError: 193291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5346.994785] Lustre: server umount lustre-MDT0001 complete [ 5367.484779] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 06:09:12 (1787738952) [ 5373.086151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5373.370148] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5373.407981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5373.588596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5377.548879] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5384.398778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5384.463305] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5384.489936] Lustre: 199594:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5388.499358] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5394.965693] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5401.095508] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5406.819761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5406.954544] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5412.775614] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5415.493301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5421.299334] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5426.235412] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5426.293504] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5432.068259] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5439.827973] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5445.612471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5445.628824] Lustre: Skipped 1 previous similar message [ 5445.633517] Lustre: lustre-OST0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 5445.644737] Lustre: Skipped 6 previous similar messages [ 5448.519557] Lustre: server umount lustre-OST0000 complete [ 5450.706553] LustreError: 201442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5450.747693] LustreError: 201442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5466.091499] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5466.096419] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5466.103408] LustreError: Skipped 1 previous similar message [ 5466.135088] Lustre: Skipped 2 previous similar messages [ 5469.628550] Lustre: server umount lustre-OST0001 complete [ 5488.095268] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5488.490707] Lustre: server umount lustre-MDT0000 complete [ 5492.950294] LustreError: 200340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787739078 with bad export cookie 3049616563210806845 [ 5492.955049] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5492.963196] LustreError: 200340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5493.486545] Lustre: server umount lustre-MDT0001 complete [ 5520.601250] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 06:11:45 (1787739105) [ 5539.424392] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 5550.113969] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5581.309537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5581.488695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5581.521465] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5581.581664] Lustre: lustre-MDT0000: new disk, initializing [ 5581.632400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5581.638233] Lustre: Skipped 3 previous similar messages [ 5581.651379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5585.473122] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5595.176559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5595.293063] Lustre: 207685:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5595.341561] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5595.344949] Lustre: Skipped 1 previous similar message [ 5595.405502] Lustre: lustre-MDT0001: new disk, initializing [ 5595.505144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5595.522038] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5598.889259] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5607.178232] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5611.644635] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5618.242943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5618.527438] Lustre: lustre-OST0000: new disk, initializing [ 5618.532974] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5618.539334] Lustre: 209653:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5624.885314] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5626.409989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5626.416499] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5626.506259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5634.979855] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5640.886649] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5641.110682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5645.870482] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5646.130747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5651.944786] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5651.958151] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5651.964779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5651.969378] Lustre: Skipped 3 previous similar messages [ 5651.981448] Lustre: Skipped 9 previous similar messages [ 5654.528764] Lustre: server umount lustre-OST0000 complete [ 5665.352513] Lustre: server umount lustre-MDT0000 complete [ 5667.311487] LustreError: 208497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5667.330571] LustreError: 208497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 5669.428597] LustreError: 207678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787739255 with bad export cookie 3049616563210808399 [ 5669.433755] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5669.437839] LustreError: 207678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5670.082570] Lustre: server umount lustre-MDT0001 complete [ 5678.884520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5684.322662] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5692.906082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5698.025961] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5705.112163] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5710.325550] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5716.705911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5716.934234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5716.939417] Lustre: Skipped 4 previous similar messages [ 5722.666669] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5729.445421] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5733.898537] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5734.086199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5738.762261] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5739.056638] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5744.609138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5744.615441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5744.627370] LustreError: Skipped 1 previous similar message [ 5744.645093] Lustre: Skipped 3 previous similar messages [ 5747.361497] Lustre: server umount lustre-OST0000 complete [ 5753.365431] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5758.104634] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5788.176256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5796.232538] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5804.471499] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5805.908125] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5809.067122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.087123] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.094133] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5814.247101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5814.253768] LustreError: Skipped 1 previous similar message [ 5814.254433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5814.266384] Lustre: Skipped 5 previous similar messages [ 5816.226754] Lustre: server umount lustre-OST0000 complete [ 5825.972959] Lustre: server umount lustre-MDT0000 complete [ 5829.604353] LustreError: 211868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5829.636985] LustreError: 211868:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5830.225342] LustreError: 211854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787739416 with bad export cookie 3049616563210809288 [ 5830.248660] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5830.757088] Lustre: server umount lustre-MDT0001 complete [ 5839.064517] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 06:17:04 (1787739424) [ 5847.064192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5852.547284] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5861.429673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5865.968568] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5872.987845] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5878.028935] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5884.665663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5891.315236] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5898.786236] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5903.059830] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5903.252130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5906.931670] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5907.203531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5913.448705] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5913.823744] Lustre: lustre-OST0001: new disk, initializing [ 5913.846537] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5913.861343] Lustre: 221205:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5919.374367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5919.387686] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5919.740155] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5923.923908] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5935.667628] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5942.414345] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5942.764701] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5950.888216] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5951.219509] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5955.581743] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5955.611779] Lustre: Skipped 5 previous similar messages [ 5955.616187] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5955.656393] LustreError: Skipped 2 previous similar messages [ 5968.351356] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5968.497284] Lustre: server umount lustre-OST0001 complete [ 5976.206318] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5976.447768] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5981.557692] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5981.847644] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6014.465925] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6014.938132] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6014.962412] Lustre: Skipped 5 previous similar messages [ 6020.095124] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6020.122717] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6020.140462] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6020.151470] Lustre: Skipped 1 previous similar message [ 6023.398506] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6031.471170] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6033.489493] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6035.437458] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6041.835878] Lustre: server umount lustre-OST0001 complete [ 6065.002847] Lustre: server umount lustre-OST0000 complete [ 6071.267843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6071.272047] Lustre: Skipped 15 previous similar messages [ 6075.786608] Lustre: server umount lustre-MDT0000 complete [ 6079.819264] LustreError: 217818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787739665 with bad export cookie 3049616563210810254 [ 6079.824441] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6079.831600] LustreError: 217818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6094.303375] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6094.652417] Lustre: server umount lustre-MDT0001 complete [ 6121.266507] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 06:21:46 (1787739706) [ 6127.827650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6128.198844] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6128.225853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6134.132657] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6144.474293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6144.646227] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6144.702087] Lustre: 226457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6151.113067] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6157.671722] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6163.815744] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6171.670435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6171.838976] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6179.554903] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6187.710147] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6196.959660] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6197.136501] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6203.581160] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6211.168962] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6212.656642] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6217.608186] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6217.955715] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6222.159589] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6222.356493] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6227.579506] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9578075ff800.ost_server_uuid 50 [ 6228.974237] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9578075ff800.ost_server_uuid in IDLE state after 0 sec [ 6229.935595] Lustre: Permanently deactivating lustre-OST0001 [ 6229.950260] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6233.548634] Lustre: Permanently reactivating lustre-OST0001 [ 6241.772735] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6241.776185] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6241.801847] Lustre: Skipped 6 previous similar messages [ 6241.823694] LustreError: Skipped 3 previous similar messages [ 6243.331427] Lustre: server umount lustre-OST0001 complete [ 6246.892607] LustreError: 229977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6246.911826] LustreError: 229977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 39 previous similar messages [ 6253.359552] Lustre: server umount lustre-OST0000 complete [ 6263.584605] Lustre: server umount lustre-MDT0000 complete [ 6267.392992] LustreError: 229330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787739853 with bad export cookie 3049616563210811542 [ 6267.395594] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6267.413658] LustreError: 229330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6267.814912] Lustre: server umount lustre-MDT0001 complete [ 6290.442698] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 06:24:35 (1787739875) [ 6298.323925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6298.569223] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6298.593320] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6298.605684] Lustre: Skipped 5 previous similar messages [ 6303.291904] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6311.204860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6311.285554] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6311.306417] Lustre: 233150:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6315.827597] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6326.607345] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6333.278367] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6339.937175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6340.263114] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6340.309579] Lustre: Permanently deactivating lustre-OST0000 [ 6340.334973] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6347.579938] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6349.890577] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6349.955378] LustreError: 235493:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -108 [ 6356.663342] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6364.069270] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6364.186349] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6372.743897] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6374.893874] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6385.702834] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6388.442475] Lustre: Permanently reactivating lustre-OST0000 [ 6388.480434] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6388.502238] Lustre: Skipped 2 previous similar messages [ 6396.717937] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6396.721592] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 47 seconds. I think it's dead, and I am evicting it. exp 00000000d73df873, cur 1787739982 deadline 1787739965 last 1787739935 [ 6396.734292] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6396.776243] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6396.919077] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6413.835075] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6413.870679] LustreError: Skipped 4 previous similar messages [ 6416.516521] Lustre: server umount lustre-OST0001 complete [ 6435.295198] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6435.682441] Lustre: server umount lustre-OST0000 complete [ 6445.885740] Lustre: server umount lustre-MDT0000 complete [ 6450.170972] LustreError: 233141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787740036 with bad export cookie 3049616563210813187 [ 6450.172378] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6450.181389] LustreError: 233141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6450.648139] Lustre: server umount lustre-MDT0001 complete [ 6465.411365] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6469.371313] Key type lgssc unregistered [ 6469.677474] LNet: 238039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6469.684719] LNetError: 238039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6470.760933] LNet: Removed LNI 192.168.201.137@tcp [ 6472.146610] Key type .llcrypt unregistered [ 6472.149653] Key type ._llcrypt unregistered [ 6496.875581] Key type ._llcrypt registered [ 6496.878648] Key type .llcrypt registered [ 6496.993968] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6498.121815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6498.130635] alg: No test for adler32 (adler32-zlib) [ 6499.180969] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 6499.583428] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6501.327148] Key type lgssc registered [ 6503.096853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6514.168936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6514.212252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6516.018942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6521.710902] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6525.416554] LustreError: 239508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6525.443291] LustreError: 239508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6530.548000] LustreError: 239509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6531.419848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6531.487930] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6531.501612] Lustre: 239497:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6531.509785] Lustre: Permanently deactivating lustre-MDT0001 [ 6531.525391] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6531.928839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6536.011310] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6539.781389] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6543.552856] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6549.691497] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6558.371282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6559.088739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6560.140166] LustreError: 241350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6560.179989] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6565.344671] LustreError: 241414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6568.953076] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6570.468497] LustreError: 241352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6576.621749] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 6578.868695] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6581.760121] LustreError: 241352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6581.777162] LustreError: 241352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6585.553990] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6585.739434] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6590.975297] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6590.983619] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6594.338095] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6602.786845] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6605.271386] Lustre: Permanently reactivating lustre-MDT0001 [ 6605.305182] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6605.316675] Lustre: Skipped 2 previous similar messages [ 6612.978031] Lustre: lustre-MDT0001: Client 989ff998-38a7-48ab-a1e9-a8a02074f89b (at 192.168.201.37@tcp) reconnecting [ 6615.184254] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6615.193697] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 6615.202852] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6615.210518] LustreError: 243163:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 6615.214723] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6615.223148] LustreError: 243163:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 6615.240909] LustreError: 243163:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.137@tcp: cfg command failed: rc = -108 [ 6615.247819] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 6615.247819] [ 6615.255058] LustreError: 239499:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6619.550237] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6621.710708] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6629.164126] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6629.631733] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6636.222928] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6636.618382] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6639.608784] Lustre: Permanently deactivating lustre-MDT0001 [ 6639.625430] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6639.632947] Lustre: Skipped 2 previous similar messages [ 6647.489831] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6657.515245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6657.517820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6657.543133] Lustre: Skipped 3 previous similar messages [ 6657.566414] Lustre: Skipped 3 previous similar messages [ 6661.581339] Lustre: server umount lustre-MDT0000 complete [ 6662.635681] LustreError: 239509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6662.667834] LustreError: 239509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6665.621292] LustreError: 239490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787740251 with bad export cookie 8765933613096180712 [ 6665.636190] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6666.284464] Lustre: server umount lustre-MDT0001 complete [ 6672.366067] Lustre: server umount lustre-OST0000 complete [ 6685.024059] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 06:31:09 (1787740269) [ 6696.167169] Lustre: server umount lustre-OST0001 complete [ 6703.493860] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 6715.266933] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6751.870280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6752.084778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6752.095985] Lustre: Skipped 2 previous similar messages [ 6752.120337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6752.195694] Lustre: lustre-MDT0000: new disk, initializing [ 6752.300701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6752.333360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6756.669885] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6768.556713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6768.793230] Lustre: 247769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6768.912875] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6768.924197] Lustre: Skipped 1 previous similar message [ 6769.048919] Lustre: lustre-MDT0001: new disk, initializing [ 6769.260896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6769.328616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6769.343230] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6774.831338] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6787.166223] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6795.478920] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6803.404212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6803.745906] Lustre: lustre-OST0000: new disk, initializing [ 6803.753879] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6803.769228] Lustre: 249741:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6803.918681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6810.494680] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6811.240823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6811.246186] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6811.498828] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6822.266739] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6824.876920] LustreError: 249999:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6834.954362] LustreError: 249999:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6834.997652] LustreError: 247779:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6835.698548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6836.014549] Lustre: lustre-OST0001: new disk, initializing [ 6836.022223] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6836.036485] Lustre: 250858:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6836.259054] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6844.578468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6844.601143] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6844.806091] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6844.876546] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6845.086083] LustreError: 247779:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6855.817463] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6857.903993] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6860.261569] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6860.262929] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6860.272617] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6860.276740] LustreError: Skipped 1 previous similar message [ 6860.316721] Lustre: Skipped 1 previous similar message [ 6865.378189] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6865.398856] Lustre: Skipped 2 previous similar messages [ 6866.313220] Lustre: server umount lustre-OST0001 complete [ 6870.505847] LustreError: 249725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6870.522110] LustreError: 249725:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6885.858126] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6885.862689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6885.869038] Lustre: Skipped 1 previous similar message [ 6890.986026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6891.020350] Lustre: Skipped 2 previous similar messages [ 6899.167176] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6899.508366] Lustre: server umount lustre-OST0000 complete [ 6906.340564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6906.365624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6906.381586] Lustre: Skipped 3 previous similar messages [ 6910.840812] Lustre: server umount lustre-MDT0000 complete [ 6915.605640] LustreError: 247764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787740501 with bad export cookie 8765933613096183750 [ 6915.614267] LustreError: 247764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6915.618289] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6922.110693] Lustre: server umount lustre-MDT0001 complete [ 6937.987922] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6942.589825] Key type lgssc unregistered [ 6942.828933] LNet: 252896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6942.836371] LNetError: 252896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6942.855977] LNet: Removed LNI 192.168.201.137@tcp [ 6943.562995] Key type .llcrypt unregistered [ 6943.565710] Key type ._llcrypt unregistered [ 6985.080937] Key type ._llcrypt registered [ 6985.094697] Key type .llcrypt registered [ 6985.250423] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6987.797946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6987.822280] alg: No test for adler32 (adler32-zlib) [ 6989.180884] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 6989.575917] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6991.523791] Key type lgssc registered [ 6992.439908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7002.554972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7002.609179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7003.837819] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7003.847891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7003.933534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7008.117053] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7017.428171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7017.548813] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7017.561553] Lustre: 255109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7018.006121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7024.203246] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7033.106873] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7039.780201] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7046.610607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7046.757227] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7046.920957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7053.378382] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7057.175362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7060.890986] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7118.314859] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7118.337996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7118.366225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7118.818318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7120.943158] Lustre: server umount lustre-OST0000 complete [ 7125.812435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7137.271737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7137.574082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7138.608892] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7138.627077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7138.629335] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7138.644419] Lustre: Skipped 1 previous similar message [ 7143.946378] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7151.037317] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7189.998047] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7190.006887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7190.010687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7190.010696] Lustre: Skipped 1 previous similar message [ 7190.016674] LustreError: Skipped 1 previous similar message [ 7190.073929] Lustre: Skipped 2 previous similar messages [ 7195.108615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7195.112717] Lustre: Skipped 2 previous similar messages [ 7195.933705] Lustre: server umount lustre-OST0000 complete [ 7202.273775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7202.293409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7202.326566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7205.348096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7207.582968] Lustre: server umount lustre-MDT0000 complete [ 7212.071546] LustreError: 256982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787740798 with bad export cookie 18272972556542723195 [ 7212.076894] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7212.090385] LustreError: 256982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7212.698953] Lustre: server umount lustre-MDT0001 complete [ 7225.766778] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 7230.461590] Key type lgssc unregistered [ 7230.895323] LNet: 260542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7230.908876] LNetError: 260542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7230.928837] LNet: Removed LNI 192.168.201.137@tcp [ 7232.254134] Key type .llcrypt unregistered [ 7232.255810] Key type ._llcrypt unregistered [ 7261.025529] Key type ._llcrypt registered [ 7261.027861] Key type .llcrypt registered [ 7261.099575] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7262.290376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7262.298072] alg: No test for adler32 (adler32-zlib) [ 7263.390289] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 7263.728197] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 7265.521144] Key type lgssc registered [ 7266.443932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7275.294553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7275.342479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7276.847323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7282.310597] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7286.248250] LustreError: 261959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7286.270859] LustreError: 261959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7291.365603] LustreError: 261960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7292.597978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7293.084252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7298.707636] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7306.064828] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7314.517098] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7323.557206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7324.004676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7330.432972] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7339.636132] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7347.570029] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7383.524514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7383.533661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7383.540782] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7383.559456] Lustre: Skipped 1 previous similar message [ 7388.633501] Lustre: server umount lustre-OST0000 complete [ 7393.203845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7393.222553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7393.225266] Lustre: Skipped 1 previous similar message [ 7398.371150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7399.534852] Lustre: server umount lustre-MDT0000 complete [ 7403.491900] LustreError: 261960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7403.537472] LustreError: 261960:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7404.461092] LustreError: 261940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787740990 with bad export cookie 13056351372715948497 [ 7404.466104] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7404.477684] LustreError: 261940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7404.922527] Lustre: server umount lustre-MDT0001 complete [ 7421.802619] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 7426.823542] Key type lgssc unregistered [ 7427.169734] LNet: 266294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7427.191899] LNetError: 266294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7427.268434] LNet: Removed LNI 192.168.201.137@tcp [ 7429.615858] Key type .llcrypt unregistered [ 7429.624201] Key type ._llcrypt unregistered [ 7453.289840] Key type ._llcrypt registered [ 7453.292352] Key type .llcrypt registered [ 7453.409944] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7454.608136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7454.691800] alg: No test for adler32 (adler32-zlib) [ 7456.294474] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 7456.725975] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 7458.472399] Key type lgssc registered [ 7460.149676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7470.089285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7470.118884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7471.674114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7476.891340] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7481.828463] LustreError: 267520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7481.860065] LustreError: 267520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7485.740253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7486.458216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7492.616192] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7502.656948] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7508.629172] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7515.742357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7516.128323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7517.178314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7523.050631] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7531.677808] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7547.874779] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7547.884974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7547.919196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7553.003269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7553.016776] Lustre: Skipped 1 previous similar message [ 7558.119726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7558.121513] Lustre: Skipped 1 previous similar message [ 7560.159127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7560.338413] Lustre: server umount lustre-OST0000 complete [ 7568.353575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7568.361505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7568.385440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7568.392910] Lustre: Skipped 2 previous similar messages [ 7571.208361] Lustre: server umount lustre-MDT0000 complete [ 7573.480857] LustreError: 268249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7573.497508] LustreError: 268249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7574.694544] LustreError: 267499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787741160 with bad export cookie 3940831181031081208 [ 7574.697817] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7574.712079] LustreError: 267499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7575.037952] Lustre: server umount lustre-MDT0001 complete [ 7586.485688] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 7589.785380] Key type lgssc unregistered [ 7590.046746] LNet: 271204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7590.061375] LNetError: 271204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7590.080902] LNet: Removed LNI 192.168.201.137@tcp [ 7591.156322] Key type .llcrypt unregistered [ 7591.162052] Key type ._llcrypt unregistered [ 7619.321454] Key type ._llcrypt registered [ 7619.323381] Key type .llcrypt registered [ 7619.465280] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7620.808763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7620.844414] alg: No test for adler32 (adler32-zlib) [ 7621.989877] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 7622.297515] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 7624.095349] Key type lgssc registered [ 7625.259800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7632.700171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7632.718789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7634.068037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7638.914534] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7644.642310] LustreError: 272625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7644.662113] LustreError: 272625:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7649.203029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7649.673030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7655.951682] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7664.275852] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7671.208332] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7680.720805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7681.317515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7688.964493] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7696.866795] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7704.886974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7735.785734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7735.795798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7735.808024] LustreError: Skipped 1 previous similar message [ 7735.809561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7735.851960] Lustre: Skipped 1 previous similar message [ 7740.277699] Lustre: server umount lustre-OST0000 complete [ 7746.017923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7746.020179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7746.034780] Lustre: Skipped 1 previous similar message [ 7746.044457] Lustre: Skipped 2 previous similar messages [ 7751.145141] LustreError: 272625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7751.353931] Lustre: server umount lustre-MDT0000 complete [ 7755.198913] LustreError: 272604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787741341 with bad export cookie 12421998055808002067 [ 7755.204838] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7755.209785] LustreError: 272604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7755.513692] Lustre: server umount lustre-MDT0001 complete [ 7766.990332] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 7770.442702] Key type lgssc unregistered [ 7770.750690] LNet: 276956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7770.757654] LNetError: 276956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7771.816339] LNet: Removed LNI 192.168.201.137@tcp [ 7772.741857] Key type .llcrypt unregistered [ 7772.744660] Key type ._llcrypt unregistered [ 7792.739519] Key type ._llcrypt registered [ 7792.744416] Key type .llcrypt registered [ 7792.951895] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7794.484976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7794.701201] alg: No test for adler32 (adler32-zlib) [ 7795.867542] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 7796.172628] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 7797.903661] Key type lgssc registered [ 7799.219241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7807.880679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7807.930393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7809.582264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7814.959801] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7819.748384] LustreError: 278193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7819.766952] LustreError: 278193:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7824.865621] LustreError: 278195:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7827.744754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7828.044911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7834.176689] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7846.459720] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7855.330541] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7866.060022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7866.800397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7873.334467] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7879.120172] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7883.237869] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7899.618260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7899.621818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7899.665927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7899.679965] Lustre: Skipped 1 previous similar message [ 7904.758511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7904.770737] Lustre: Skipped 1 previous similar message [ 7906.027850] Lustre: server umount lustre-OST0000 complete [ 7914.977342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7914.983411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7914.988304] Lustre: Skipped 2 previous similar messages [ 7914.988677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7915.000919] Lustre: Skipped 1 previous similar message [ 7918.686871] Lustre: server umount lustre-MDT0000 complete [ 7920.099888] LustreError: 278203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7920.120031] LustreError: 278203:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7922.996664] LustreError: 278158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787741508 with bad export cookie 8577353760007399489 [ 7923.009329] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7923.017532] LustreError: 278158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7923.606212] Lustre: server umount lustre-MDT0001 complete [ 7936.051558] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 7940.247973] Key type lgssc unregistered [ 7940.641058] LNet: 281903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7940.650275] LNetError: 281903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7940.688167] LNet: Removed LNI 192.168.201.137@tcp [ 7942.009859] Key type .llcrypt unregistered [ 7942.012908] Key type ._llcrypt unregistered [ 7960.121392] Key type ._llcrypt registered [ 7960.124078] Key type .llcrypt registered [ 7960.320406] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7979.615235] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 7994.685581] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7996.123439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7996.187787] alg: No test for adler32 (adler32-zlib) [ 7997.349170] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 7997.659281] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 7999.330954] Key type lgssc registered [ 8000.733578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8036.452630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8036.480346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8037.899768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8038.053124] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8038.125232] Lustre: lustre-MDT0000: new disk, initializing [ 8038.178504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8038.200946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8044.003190] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8056.240463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8056.380418] Lustre: 285457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8056.449760] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8056.454280] Lustre: Skipped 1 previous similar message [ 8056.563442] Lustre: lustre-MDT0001: new disk, initializing [ 8056.691044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8056.747801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8056.757881] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8062.272832] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8074.129531] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8081.806270] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8088.748786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8088.914690] Lustre: lustre-OST0000: new disk, initializing [ 8088.919511] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8088.929697] Lustre: 287429:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8088.990140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8095.659204] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8096.817087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8096.821279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8096.848905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8108.536936] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8114.146509] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8114.417592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8119.782402] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8120.023819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8122.344043] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8122.345739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8122.353360] LustreError: Skipped 1 previous similar message [ 8122.378570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8122.397365] Lustre: Skipped 1 previous similar message [ 8127.465636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8127.475859] Lustre: Skipped 1 previous similar message [ 8128.566913] Lustre: server umount lustre-OST0000 complete [ 8133.604091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8133.618276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8133.635108] Lustre: Skipped 1 previous similar message [ 8133.653361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8137.709471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8137.721501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8137.730349] Lustre: Skipped 1 previous similar message [ 8139.197653] Lustre: server umount lustre-MDT0000 complete [ 8143.931571] LustreError: 285450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787741729 with bad export cookie 10240376241847458673 [ 8143.933460] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8143.945419] LustreError: 285450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8144.312312] Lustre: server umount lustre-MDT0001 complete [ 8154.081572] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 06:55:38 (1787741738) [ 8161.149967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8161.840971] LustreError: 289841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8161.857321] LustreError: 289841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8161.918283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8166.773683] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8167.399960] LustreError: 289842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8171.497974] LustreError: 289841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8176.036301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8176.252663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8181.253606] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8190.702951] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8197.714663] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8205.062260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8205.291446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8211.491042] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8221.858983] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8237.030529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8237.031890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8237.032183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8237.040859] Lustre: Skipped 1 previous similar message [ 8241.257091] Lustre: server umount lustre-OST0000 complete [ 8247.271220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8247.290919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8247.349920] Lustre: Skipped 1 previous similar message [ 8247.386134] Lustre: Skipped 2 previous similar messages [ 8252.391638] LustreError: 289841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8252.407738] LustreError: 289841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8252.438856] Lustre: server umount lustre-MDT0000 complete [ 8257.236437] LustreError: 289822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787741843 with bad export cookie 10240376241847459555 [ 8257.248486] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8257.254745] LustreError: 289822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8257.835979] Lustre: server umount lustre-MDT0001 complete [ 8271.737995] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 8274.536577] Key type lgssc unregistered [ 8274.865236] LNet: 293403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8274.871324] LNetError: 293403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8274.886931] LNet: Removed LNI 192.168.201.137@tcp [ 8275.905739] Key type .llcrypt unregistered [ 8275.907945] Key type ._llcrypt unregistered [ 8310.283333] Key type ._llcrypt registered [ 8310.286267] Key type .llcrypt registered [ 8310.524200] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 8331.884843] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 8333.602790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8333.954171] alg: No test for adler32 (adler32-zlib) [ 8335.701692] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 8336.139230] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 8337.839147] Key type lgssc registered [ 8339.347737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8394.066846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8394.135697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8395.478812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8395.501799] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8395.582549] Lustre: lustre-MDT0000: new disk, initializing [ 8395.781911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8395.858848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8401.886793] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8412.628069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8412.775426] Lustre: 297231:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8412.859857] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8412.866515] Lustre: Skipped 1 previous similar message [ 8412.985806] Lustre: lustre-MDT0001: new disk, initializing [ 8413.095206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8413.141381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8413.146503] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8417.921762] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8428.505966] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8435.929633] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8445.219150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8445.877461] Lustre: lustre-OST2710: new disk, initializing [ 8445.886664] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8445.891563] Lustre: 299197:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8446.019357] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8451.674888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8451.685153] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8451.807644] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8453.752547] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8465.783849] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8474.828288] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8475.204516] Lustre: lustre-OST03e8: new disk, initializing [ 8475.210371] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8475.224348] Lustre: 300266:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8475.388252] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8480.911649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8480.937591] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8481.064437] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8483.365269] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8492.200324] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8501.510908] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8501.800667] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8507.767241] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8508.073254] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8513.518855] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8513.850599] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8519.368647] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8519.709782] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8527.336485] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8527.351627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8527.362034] Lustre: Skipped 3 previous similar messages [ 8527.387285] Lustre: Skipped 3 previous similar messages [ 8532.289815] Lustre: server umount lustre-MDT0000 complete [ 8535.700929] LustreError: 297224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787742121 with bad export cookie 7815211865341219440 [ 8535.712512] LustreError: 297224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8535.716540] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8536.147045] Lustre: server umount lustre-MDT0001 complete [ 8550.783109] Lustre: server umount lustre-OST2710 complete [ 8553.952064] Lustre: 295028:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787742123/real 1787742123] req@00000000d7ea3a1f x1874583273677184/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787742139 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8553.992302] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8556.166907] Lustre: server umount lustre-OST03e8 complete [ 8580.061788] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 8590.990989] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 8630.861729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8631.206912] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8631.249481] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8631.338749] Lustre: lustre-MDT0000: new disk, initializing [ 8631.394363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8631.416195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8636.241698] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8647.791889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8648.046376] Lustre: 305532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8648.132670] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8648.136570] Lustre: Skipped 1 previous similar message [ 8648.213574] Lustre: lustre-MDT0001: new disk, initializing [ 8648.337574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8648.367256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8648.373300] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8652.790953] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8664.394375] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8672.613313] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8682.703376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8682.978402] Lustre: lustre-OST0000: new disk, initializing [ 8682.982435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8682.990921] Lustre: 307502:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8683.120245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8690.229156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8690.238946] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8690.429045] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8691.517354] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8701.617292] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8707.523850] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8707.861604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8714.092404] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8714.375785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8721.380574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8721.384229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8721.394041] LustreError: Skipped 1 previous similar message [ 8721.404574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8721.413383] Lustre: Skipped 1 previous similar message [ 8721.428793] Lustre: Skipped 1 previous similar message [ 8722.640285] Lustre: server umount lustre-OST0000 complete [ 8726.498983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8726.502088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8726.509457] Lustre: Skipped 1 previous similar message [ 8726.524592] Lustre: Skipped 1 previous similar message [ 8731.620553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8731.632459] Lustre: Skipped 1 previous similar message [ 8732.697418] Lustre: server umount lustre-MDT0000 complete [ 8736.737133] LustreError: 307699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8736.762266] LustreError: 307699:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8737.544374] LustreError: 307501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787742323 with bad export cookie 7815211865341221134 [ 8737.546636] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8737.563673] LustreError: 307501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8738.086368] Lustre: server umount lustre-MDT0001 complete [ 8748.753117] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 07:05:33 (1787742333) [ 8750.465671] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8753.161461] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 07:05:37 (1787742337) [ 8758.451197] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 8792.428251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8792.630885] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8792.647298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8792.862289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8797.633279] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8809.000674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8809.173157] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8809.193866] Lustre: 311363:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8815.334808] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8839.862475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8840.169918] Lustre: Denying initial registration attempt from nid 192.168.201.137@tcp, specified as failover [ 8840.187572] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8840.208283] LustreError: 312816:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 8840.222812] LustreError: 312816:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8840.235230] LustreError: 312816:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8840.366280] Lustre: server umount lustre-OST0000 complete [ 8840.368892] LustreError: 312816:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8850.400353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8850.406133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8850.418256] Lustre: Skipped 1 previous similar message [ 8850.443410] Lustre: Skipped 1 previous similar message [ 8854.626423] Lustre: server umount lustre-MDT0000 complete [ 8855.529648] LustreError: 312818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8855.557373] LustreError: 312818:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8859.533699] LustreError: 311356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787742445 with bad export cookie 7815211865341222016 [ 8859.564723] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8859.567793] LustreError: 311356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8860.422980] Lustre: server umount lustre-MDT0001 complete [ 8874.016673] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 8877.380188] Key type lgssc unregistered [ 8877.526748] LNet: 313943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8877.536083] LNetError: 313943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8878.566986] LNet: Removed LNI 192.168.201.137@tcp [ 8879.504222] Key type .llcrypt unregistered [ 8879.505776] Key type ._llcrypt unregistered [ 8895.257545] Key type ._llcrypt registered [ 8895.259869] Key type .llcrypt registered [ 8895.339717] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 8896.482214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8896.605353] alg: No test for adler32 (adler32-zlib) [ 8897.957202] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 8898.356912] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 8900.135192] Key type lgssc registered [ 8901.692383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8954.563539] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 8963.578193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8963.598038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8964.928944] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8964.976157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8965.169348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8970.005944] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8979.239441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8979.303753] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8979.348920] Lustre: 316715:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8979.581985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8984.382706] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8991.855842] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8998.042214] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9019.246777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9019.547598] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9019.754703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9028.060876] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9036.614387] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9039.331594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9039.333026] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9039.346660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9039.351464] Lustre: Skipped 1 previous similar message [ 9039.363495] Lustre: Skipped 1 previous similar message [ 9044.449786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9044.457424] Lustre: Skipped 1 previous similar message [ 9049.568368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9049.574247] Lustre: Skipped 1 previous similar message [ 9053.151680] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9053.431442] Lustre: server umount lustre-OST0000 complete [ 9059.811842] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9059.815191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9059.824261] Lustre: Skipped 1 previous similar message [ 9059.832667] Lustre: Skipped 1 previous similar message [ 9063.612555] Lustre: server umount lustre-MDT0000 complete [ 9067.468822] LustreError: 318579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787742653 with bad export cookie 7866258890713279568 [ 9067.474210] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9067.482480] LustreError: 318579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9067.770383] Lustre: server umount lustre-MDT0001 complete [ 9080.629820] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 9083.814722] Key type lgssc unregistered [ 9084.145228] LNet: 320291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9084.166778] LNetError: 320291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9084.176922] LNet: Removed LNI 192.168.201.137@tcp [ 9085.178403] Key type .llcrypt unregistered [ 9085.180435] Key type ._llcrypt unregistered [ 9111.733818] Key type ._llcrypt registered [ 9111.735901] Key type .llcrypt registered [ 9111.827477] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 9113.360139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9113.385512] alg: No test for adler32 (adler32-zlib) [ 9114.516983] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 9114.776746] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 9116.425625] Key type lgssc registered [ 9117.973831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9127.069503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9127.143610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9128.792068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9133.540626] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9138.161755] LustreError: 321708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9138.196858] LustreError: 321708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9142.403666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9142.916483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9148.673748] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9157.519254] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9163.463647] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9172.326061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9172.728444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9179.092391] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9188.499982] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9199.072509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9199.084950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9199.106374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9200.625118] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 9202.661192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9202.662159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9202.716655] Lustre: Skipped 2 previous similar messages [ 9204.137682] Lustre: server umount lustre-MDT0000 complete [ 9205.721221] LustreError: 324290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9207.779912] LustreError: 321704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9207.807031] LustreError: 321704:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9209.204280] LustreError: 321689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787742795 with bad export cookie 3310082308300888667 [ 9209.207563] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9209.212774] LustreError: 321689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9210.840206] LustreError: 321703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9210.881121] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [ 9212.897540] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9212.915513] Lustre: Skipped 1 previous similar message [ 9214.956558] LustreError: 321705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9214.966181] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9214.979750] LustreError: 321705:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9215.004426] Lustre: Skipped 1 previous similar message [ 9215.949864] Lustre: server umount lustre-MDT0001 complete [ 9224.966993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9235.308768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9254.884414] LustreError: 321018:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000011a63486 x1874584089429632/t0(0) o250->MGC192.168.201.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9255.315296] LustreError: 323542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9255.434092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9255.495511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9259.922145] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9269.307824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9269.739547] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9269.906454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9274.924641] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9275.390268] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9286.064864] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9288.260700] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9295.559344] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9297.085587] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9301.474858] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9301.481664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9301.496300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9305.570507] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9305.577840] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9307.544437] Lustre: server umount lustre-OST0000 complete [ 9315.812191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9315.821090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9320.011540] Lustre: server umount lustre-MDT0000 complete [ 9321.440778] LustreError: 325312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9321.487519] LustreError: 325312:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9325.282194] LustreError: 322442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787742911 with bad export cookie 3310082308300895842 [ 9325.282631] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9325.287965] LustreError: 322442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9325.720770] Lustre: server umount lustre-MDT0001 complete [ 9339.044959] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 9342.670784] Key type lgssc unregistered [ 9343.069820] LNet: 327987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9343.080450] LNetError: 327987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9343.101055] LNet: Removed LNI 192.168.201.137@tcp [ 9344.520103] Key type .llcrypt unregistered [ 9344.524966] Key type ._llcrypt unregistered [ 9370.588000] Key type ._llcrypt registered [ 9370.590123] Key type .llcrypt registered [ 9370.840831] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 9373.951441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9374.042962] alg: No test for adler32 (adler32-zlib) [ 9375.154554] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 9375.447156] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 9377.203666] Key type lgssc registered [ 9379.358779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9388.198392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9388.223079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9389.719466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9394.613285] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9399.272907] LustreError: 329404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9399.297498] LustreError: 329404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9403.397630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9404.177805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9411.977977] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9422.829923] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9429.567682] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9434.083272] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9434.088107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9434.106787] Lustre: Skipped 1 previous similar message [ 9434.121568] Lustre: Skipped 1 previous similar message [ 9438.712908] Lustre: server umount lustre-MDT0000 complete [ 9439.202624] LustreError: 329405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9439.216223] LustreError: 329405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9442.406106] LustreError: 329386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787743028 with bad export cookie 9694362380033727406 [ 9442.411828] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9442.414465] LustreError: 329386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9442.716489] Lustre: server umount lustre-MDT0001 complete [ 9455.837075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9456.013886] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9456.034298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9456.316675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9461.147589] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9470.216720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9470.325492] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9470.380381] Lustre: 331989:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9470.440989] LustreError: 332002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9470.688894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9475.484387] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9482.964472] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9488.651893] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9499.016061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9499.371307] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9499.388952] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9499.397610] LustreError: 333878:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 9499.414399] LustreError: 333878:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9499.422929] LustreError: 333878:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9499.536533] Lustre: server umount lustre-OST0000 complete [ 9499.538672] LustreError: 333878:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9510.132026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9510.391702] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9510.608502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9516.185138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9517.083406] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9524.503238] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9533.129170] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9533.271862] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9533.344403] Lustre: lustre-OST0001: new disk, initializing [ 9533.349156] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9533.359121] Lustre: 335417:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9533.489689] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9539.224617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9539.244128] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9539.376112] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9540.264185] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9547.757190] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9554.912633] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9554.918056] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9554.936503] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9554.966129] Lustre: Skipped 1 previous similar message [ 9556.659396] Lustre: server umount lustre-OST0001 complete [ 9560.033928] LustreError: 334346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9560.040853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9560.052337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9560.052347] Lustre: Skipped 1 previous similar message [ 9560.058544] LustreError: 334346:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9560.095918] Lustre: Skipped 1 previous similar message [ 9565.159611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9565.161438] Lustre: Skipped 1 previous similar message [ 9565.163844] LustreError: 334343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9565.183901] LustreError: 334343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9570.281630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9570.290747] Lustre: Skipped 1 previous similar message [ 9574.368642] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9574.608739] Lustre: server umount lustre-OST0000 complete [ 9578.471548] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9578.476205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9578.484898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9584.733740] Lustre: server umount lustre-MDT0000 complete [ 9585.639503] LustreError: 333025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9585.665971] LustreError: 333025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9588.461460] LustreError: 332730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787743174 with bad export cookie 9694362380033727966 [ 9588.465416] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9588.472737] LustreError: 332730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9588.807373] Lustre: server umount lustre-MDT0001 complete [ 9598.703943] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 9601.396351] Key type lgssc unregistered [ 9601.651947] LNet: 337381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9601.661598] LNetError: 337381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9601.682618] LNet: Removed LNI 192.168.201.137@tcp [ 9602.671528] Key type .llcrypt unregistered [ 9602.673933] Key type ._llcrypt unregistered [ 9685.515935] Key type ._llcrypt registered [ 9685.518191] Key type .llcrypt registered [ 9685.677927] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 9699.966875] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 9701.901734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9701.924490] alg: No test for adler32 (adler32-zlib) [ 9703.182202] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [ 9703.528089] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 9705.295139] Key type lgssc registered [ 9706.861926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9748.071553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9748.127147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9749.474420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9749.525534] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9749.634187] Lustre: lustre-MDT0000: new disk, initializing [ 9749.726224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9749.757644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9755.116439] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9765.584514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9765.730511] Lustre: 342318:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9765.872345] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9765.877442] Lustre: Skipped 1 previous similar message [ 9765.958357] Lustre: lustre-MDT0001: new disk, initializing [ 9766.107678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9766.152099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9766.176083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9771.216758] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9778.773907] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9783.600916] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9791.529141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9792.009515] Lustre: lustre-OST0000: new disk, initializing [ 9792.016444] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9792.031279] Lustre: 344284:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9792.232864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9799.907665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9799.915233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9800.059399] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9802.174589] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9811.614415] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9816.054342] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9816.170240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9822.074811] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9822.598214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9830.886815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9830.894054] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9830.916438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9832.493191] Lustre: server umount lustre-OST0000 complete [ 9841.123625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9841.150924] Lustre: Skipped 1 previous similar message [ 9841.155635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9841.162186] Lustre: Skipped 2 previous similar messages [ 9844.569591] Lustre: server umount lustre-MDT0000 complete [ 9846.248583] LustreError: 344455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9846.260455] LustreError: 344455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9847.614556] LustreError: 342311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787743433 with bad export cookie 12097711940484795073 [ 9847.617255] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9847.623928] LustreError: 342311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9847.829785] Lustre: server umount lustre-MDT0001 complete [ 9858.260388] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 07:24:02 (1787743442) [ 9865.631860] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 07:24:10 (1787743450) [ 9874.995328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9875.544185] LustreError: 347044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9875.643150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9881.056500] LustreError: 347045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9882.091839] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9885.167565] LustreError: 347044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9890.298453] LustreError: 347045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9891.675654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9891.951616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9896.466242] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9904.345985] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9909.440988] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9916.744930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9917.284632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9924.865315] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9932.768214] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9945.778927] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9947.985128] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9949.955576] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9952.335201] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9954.842811] Lustre: Failing over lustre-MDT0000 [ 9955.055468] Lustre: server umount lustre-MDT0000 complete [ 9955.850141] LustreError: 348623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9957.354077] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9957.380720] Lustre: Skipped 2 previous similar messages [ 9972.233350] LustreError: 348623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9972.273702] LustreError: 348623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 9973.537744] Lustre: 340541:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787743543/real 1787743543] req@00000000933cd0ff x1874584707083520/t0(0) o400->MGC192.168.201.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787743559 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9973.549826] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9974.617845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9983.978055] LustreError: 340539:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009391ba4e x1874584707089152/t0(0) o250->MGC192.168.201.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9984.223377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9984.294403] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9987.538796] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9989.382775] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9989.628342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9989.688771] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 9989.753033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9999.120622] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10000.908870] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10004.767747] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10006.458921] Lustre: Failing over lustre-MDT0000 [10006.781417] Lustre: server umount lustre-MDT0000 complete [10008.054532] LustreError: 349536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10008.075931] LustreError: 349536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [10010.093033] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10010.095772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10010.108986] Lustre: Skipped 2 previous similar messages [10010.118906] LustreError: Skipped 1 previous similar message [10024.809776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10024.944526] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10025.357588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10025.486414] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10029.522704] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10030.593338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10030.614052] Lustre: Skipped 2 previous similar messages [10030.672982] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10030.698327] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10030.756316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10039.791899] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10041.178583] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10051.069960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10051.079669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10051.087030] Lustre: Skipped 2 previous similar messages [10051.110784] Lustre: Skipped 2 previous similar messages [10054.704740] Lustre: server umount lustre-MDT0000 complete [10059.372349] LustreError: 347024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787743645 with bad export cookie 12097711940484797782 [10059.383298] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10059.387527] LustreError: 347024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10059.843108] Lustre: server umount lustre-MDT0001 complete [10074.698714] Lustre: server umount lustre-OST0000 complete [10090.794684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10091.194581] LustreError: 353415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10091.212757] LustreError: 353415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [10091.246552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10096.885610] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10105.890366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10111.876279] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10119.796794] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10126.005876] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10133.584243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10136.240755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10141.016161] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10148.805898] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10161.129095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [10161.136355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10161.148504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10165.296901] Lustre: server umount lustre-OST0000 complete [10172.397960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10172.406686] Lustre: Skipped 3 previous similar messages [10175.385759] Lustre: server umount lustre-MDT0000 complete [10178.728346] LustreError: 353396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787743764 with bad export cookie 12097711940484798531 [10178.755432] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10178.757573] LustreError: 353396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10179.619764] Lustre: server umount lustre-MDT0001 complete [10190.036850] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [10192.385892] Key type lgssc unregistered [10192.691366] LNet: 356951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10192.696872] LNetError: 356951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10192.722242] LNet: Removed LNI 192.168.201.137@tcp [10193.591651] Key type .llcrypt unregistered [10193.595663] Key type ._llcrypt unregistered [10219.049719] Key type ._llcrypt registered [10219.054213] Key type .llcrypt registered [10219.153280] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [10220.635123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10220.678128] alg: No test for adler32 (adler32-zlib) [10221.795335] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [10222.068985] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [10223.831362] Key type lgssc registered [10225.244497] Lustre: Echo OBD driver; http://www.lustre.org/ [10232.603996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10232.619148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10234.022117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10239.441765] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10243.552025] LustreError: 358470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10243.562811] LustreError: 358470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10248.689286] LustreError: 358470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10249.364195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10249.619627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10254.279738] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10263.192239] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10270.103110] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10278.452752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10278.794927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10288.046824] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10295.466400] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10303.474206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10318.686111] Lustre: server umount lustre-MDT0000 complete [10318.824880] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10318.827918] LustreError: 358466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10318.848030] Lustre: Skipped 2 previous similar messages [10318.883882] LustreError: 358466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10322.516238] LustreError: 361074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787743908 with bad export cookie 6640290510211983994 [10322.517280] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10322.530489] LustreError: 361074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10322.813931] Lustre: server umount lustre-MDT0001 complete [10336.396588] Lustre: server umount lustre-OST0000 complete [10349.807091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10350.363619] LustreError: 362206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10350.475627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10355.447571] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10355.689994] LustreError: 362207:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10364.531180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10364.725947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10369.730472] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10377.832345] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10383.012727] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10390.104181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10390.325848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10395.472257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10397.632649] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10407.297246] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10420.196645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10420.214591] Lustre: Skipped 2 previous similar messages [10420.214241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10420.226623] Lustre: Skipped 2 previous similar messages [10425.333805] LustreError: 362207:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10425.370781] LustreError: 362207:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10425.406816] Lustre: server umount lustre-MDT0000 complete [10429.415218] LustreError: 362186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787744015 with bad export cookie 6640290510211985114 [10429.416842] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10429.429778] LustreError: 362186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10429.891437] Lustre: server umount lustre-MDT0001 complete [10444.669749] Lustre: server umount lustre-OST0000 complete [10463.406585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10463.935821] LustreError: 366116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10464.084945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10469.640348] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10478.523531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10484.013882] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10490.910548] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10497.486401] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10506.653833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10506.992991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10507.005172] Lustre: Skipped 1 previous similar message [10509.247380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10516.246974] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10527.432032] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10533.861698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10533.889140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10533.895454] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10533.896942] Lustre: Skipped 1 previous similar message [10538.981367] LustreError: 367939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10538.997131] LustreError: 367939:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10539.073067] Lustre: server umount lustre-OST0000 complete [10544.099053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10544.116733] Lustre: Skipped 1 previous similar message [10544.126228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10549.070592] Lustre: server umount lustre-MDT0000 complete [10552.143494] LustreError: 366098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787744138 with bad export cookie 6640290510211986136 [10552.153466] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10552.443209] Lustre: server umount lustre-MDT0001 complete [10564.938948] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [10570.390870] Key type lgssc unregistered [10570.920701] LNet: 369652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10570.933291] LNetError: 369652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10570.952690] LNet: Removed LNI 192.168.201.137@tcp [10572.312598] Key type .llcrypt unregistered [10572.315320] Key type ._llcrypt unregistered [10600.188405] Key type ._llcrypt registered [10600.190211] Key type .llcrypt registered [10600.318392] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [10602.186029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10602.238150] alg: No test for adler32 (adler32-zlib) [10603.630244] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [10604.197638] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [10605.943241] Key type lgssc registered [10607.194099] Lustre: Echo OBD driver; http://www.lustre.org/ [10617.867815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10617.901791] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10617.911188] LustreError: 371075:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10617.951886] LustreError: 371075:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10617.958705] LustreError: 371075:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10617.967786] LustreError: 371075:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10617.982405] LustreError: 371075:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10625.686663] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10625.690596] LustreError: 371474:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10625.699549] LustreError: 371474:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10625.702352] LustreError: 371474:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10625.704893] LustreError: 371474:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10625.707866] LustreError: 371474:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10644.442236] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [10647.925955] Key type lgssc unregistered [10648.577823] LNet: 372480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10648.597883] LNetError: 372480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10648.621443] LNet: Removed LNI 192.168.201.137@tcp [10649.738444] Key type .llcrypt unregistered [10649.740914] Key type ._llcrypt unregistered [10671.591842] Key type ._llcrypt registered [10671.596863] Key type .llcrypt registered [10671.758689] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [10688.862462] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [10690.000588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10690.101587] alg: No test for adler32 (adler32-zlib) [10691.138783] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [10691.523148] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [10693.208212] Key type lgssc registered [10695.049675] Lustre: Echo OBD driver; http://www.lustre.org/ [10737.428858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10737.452989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10738.808368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10738.846951] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10738.986656] Lustre: lustre-MDT0000: new disk, initializing [10739.156502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10739.201732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10747.958113] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10760.117942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10760.262403] Lustre: 375640:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10760.351548] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10760.357256] Lustre: Skipped 1 previous similar message [10760.420619] Lustre: lustre-MDT0001: new disk, initializing [10760.470321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10760.504723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10760.510488] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10765.744383] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10776.501431] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10783.719706] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10790.970818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10791.527749] Lustre: lustre-OST0000: new disk, initializing [10791.537261] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10791.553138] Lustre: 377610:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10791.686832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10797.175892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10797.197156] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10797.395472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10799.572850] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10809.055528] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10814.676533] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10814.815639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10818.905454] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10819.123387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10823.144163] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10823.146994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10823.155409] LustreError: Skipped 1 previous similar message [10823.157291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10823.184397] Lustre: Skipped 1 previous similar message [10827.261767] Lustre: server umount lustre-OST0000 complete [10832.352745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10832.381561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10832.398991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10832.409871] Lustre: Skipped 1 previous similar message [10837.648938] Lustre: server umount lustre-MDT0000 complete [10841.304390] LustreError: 375631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787744427 with bad export cookie 17735002132362245259 [10841.311353] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10841.314070] LustreError: 375631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10842.187957] Lustre: server umount lustre-MDT0001 complete [10852.430059] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10855.758962] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 07:40:39 (1787744439) [10865.307849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10866.439120] LustreError: 380071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10866.476140] LustreError: 380071:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10866.608470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10871.797340] LustreError: 380072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10872.254126] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10876.911442] LustreError: 380071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10882.021320] LustreError: 380072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10882.178983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10882.517809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10890.032471] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10898.366431] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10904.787155] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10912.566634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10912.877687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10921.030329] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10933.232336] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10943.738114] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10943.966650] Lustre: lustre-OST0001: new disk, initializing [10943.970528] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10943.979218] Lustre: 382908:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10944.135057] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10950.744132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10950.755376] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10950.841837] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10951.000604] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10959.861672] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10966.515037] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10966.520397] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10966.521973] Lustre: lustre-OST0001: Not available for connect from 192.168.201.37@tcp (stopping) [10966.521980] Lustre: Skipped 2 previous similar messages [10966.530638] LustreError: Skipped 1 previous similar message [10966.596186] Lustre: Skipped 2 previous similar messages [10969.414571] Lustre: server umount lustre-OST0001 complete [10971.600969] LustreError: 381893:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10981.873383] LustreError: 381891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10981.884082] LustreError: 381891:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10986.978075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10986.992293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10986.995250] Lustre: Skipped 3 previous similar messages [10992.101659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10992.127375] Lustre: Skipped 1 previous similar message [10992.633814] Lustre: server umount lustre-OST0000 complete [11000.291642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11000.305300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11000.310201] Lustre: Skipped 1 previous similar message [11000.319188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11005.803176] Lustre: server umount lustre-MDT0000 complete [11007.484851] LustreError: 380071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11007.522851] LustreError: 380071:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11011.631464] LustreError: 380053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787744597 with bad export cookie 17735002132362246148 [11011.632610] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11011.643583] LustreError: 380053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11018.066820] Lustre: server umount lustre-MDT0001 complete [11031.241630] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [11034.324827] Key type lgssc unregistered [11034.587975] LNet: 384898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11034.595167] LNetError: 384898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11034.614567] LNet: Removed LNI 192.168.201.137@tcp [11036.067587] Key type .llcrypt unregistered [11036.069312] Key type ._llcrypt unregistered [11091.923783] Key type ._llcrypt registered [11091.926775] Key type .llcrypt registered [11092.051852] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [11094.850131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11094.914098] alg: No test for adler32 (adler32-zlib) [11096.327199] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [11096.728640] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [11098.450910] Key type lgssc registered [11100.145209] Lustre: Echo OBD driver; http://www.lustre.org/ [11112.348204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11112.399738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11113.915164] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11114.024371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11114.324672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11120.110253] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11130.307216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11130.362540] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11130.385180] Lustre: 387701:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11130.598958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11135.891255] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11143.869457] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11150.745491] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11154.402267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11154.404605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11154.433846] Lustre: Skipped 1 previous similar message [11154.454020] Lustre: Skipped 1 previous similar message [11159.455710] Lustre: server umount lustre-MDT0000 complete [11163.437446] LustreError: 389292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787744749 with bad export cookie 14626574802781437655 [11163.444699] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11163.456734] LustreError: 389292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11163.769879] Lustre: server umount lustre-MDT0001 complete [11173.396621] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 07:45:58 (1787744758) [11181.398348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11181.887245] LustreError: 390210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11181.898447] LustreError: 390210:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11181.969585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11187.168979] LustreError: 390211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11189.494459] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11192.290046] LustreError: 390210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11197.412681] LustreError: 390211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11201.445555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11202.017500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11208.341784] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11217.627942] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11225.291540] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11234.901302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11235.063075] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11235.234596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11241.249796] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11249.172995] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11259.446542] Lustre: Permanently deactivating lustre-OST0000 [11259.491970] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11265.052481] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11267.594842] LustreError: 393102:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11267.599964] LustreError: 393102:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11267.603745] LustreError: 393102:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11272.981221] LustreError: 393151:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11273.024796] LustreError: 393151:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11283.234705] Lustre: server umount lustre-OST0000 complete [11286.775918] LustreError: 393401:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11286.833085] LustreError: 393401:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11292.131808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11292.136963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11292.155846] Lustre: Skipped 1 previous similar message [11292.185369] Lustre: Skipped 1 previous similar message [11297.249877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11297.254485] Lustre: Skipped 1 previous similar message [11302.368566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11302.382993] Lustre: Skipped 1 previous similar message [11303.391129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11303.681291] Lustre: server umount lustre-MDT0000 complete [11307.547701] LustreError: 390206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11307.605055] LustreError: 390206:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11308.514162] LustreError: 390192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787744894 with bad export cookie 14626574802781438236 [11308.516717] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11308.522916] LustreError: 390192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11319.104729] Lustre: server umount lustre-MDT0001 complete [11328.673390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11333.364750] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11336.269193] LustreError: 394562:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11344.144991] Lustre: 394903:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.137@tcp [11346.522677] Lustre: server umount MGS complete [11361.043119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11361.932738] LustreError: 395629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11361.969834] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11361.986890] Lustre: Skipped 1 previous similar message [11362.069899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11369.344705] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11380.732317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11381.052228] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11381.115809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11386.652554] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11395.755574] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11401.527837] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11403.476630] Lustre: Permanently reactivating lustre-OST0000 [11403.498275] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11403.504158] Lustre: Skipped 2 previous similar messages [11429.343100] LustreError: 397362:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11439.004659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11439.487622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11446.451478] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11454.584249] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11460.068074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11460.068319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11460.082433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11460.090209] Lustre: Skipped 1 previous similar message [11465.970709] Lustre: server umount lustre-OST0000 complete [11473.379526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11473.387993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11473.394780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11473.400750] Lustre: Skipped 3 previous similar messages [11485.151155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11485.487613] Lustre: server umount lustre-MDT0000 complete [11485.667755] LustreError: 395630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11485.697154] LustreError: 395630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11490.455836] LustreError: 395611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787745076 with bad export cookie 14626574802781439286 [11490.472359] LustreError: 395611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11490.472874] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11491.000059] Lustre: server umount lustre-MDT0001 complete [11506.236261] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [11509.717173] Key type lgssc unregistered [11510.081677] LNet: 399950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11510.088117] LNetError: 399950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11510.115186] LNet: Removed LNI 192.168.201.137@tcp [11511.335901] Key type .llcrypt unregistered [11511.339717] Key type ._llcrypt unregistered [11534.785984] Key type ._llcrypt registered [11534.787613] Key type .llcrypt registered [11534.894962] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [11553.567186] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [11555.837270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11555.852877] alg: No test for adler32 (adler32-zlib) [11557.119957] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [11557.454964] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [11559.151558] Key type lgssc registered [11561.086864] Lustre: Echo OBD driver; http://www.lustre.org/ [11602.874610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11602.883855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11604.130228] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11604.167292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11604.265941] Lustre: lustre-MDT0000: new disk, initializing [11604.365962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11604.401297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11610.219714] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11623.250175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11623.470879] Lustre: 403112:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11623.617816] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11623.626454] Lustre: Skipped 1 previous similar message [11623.764962] Lustre: lustre-MDT0001: new disk, initializing [11623.932927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11624.000038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11624.016945] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11629.980514] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11641.295249] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11647.513863] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11656.543197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11657.001739] Lustre: lustre-OST0000: new disk, initializing [11657.010241] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11657.016777] Lustre: 405083:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11657.082486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11664.119048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11664.159275] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11664.399909] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11668.032843] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11683.020878] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11688.733694] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11689.060266] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11695.019592] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11695.338476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11700.198300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11700.203986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11700.212711] LustreError: Skipped 1 previous similar message [11700.213454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11700.245016] Lustre: Skipped 1 previous similar message [11704.198495] Lustre: server umount lustre-OST0000 complete [11710.442294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11710.468391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11710.478840] Lustre: Skipped 2 previous similar messages [11715.556091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11720.673501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11720.682619] Lustre: Skipped 2 previous similar messages [11723.233042] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11723.660746] Lustre: server umount lustre-MDT0000 complete [11728.256585] LustreError: 403106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787745314 with bad export cookie 6278842238131083577 [11728.269339] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11728.623129] Lustre: server umount lustre-MDT0001 complete [11740.144484] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11742.530100] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 07:55:27 (1787745327) [11751.442417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11752.046917] LustreError: 407548:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11752.058512] LustreError: 407548:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11752.158382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11757.536715] LustreError: 407549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11757.986761] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11762.674222] LustreError: 407548:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11767.778442] LustreError: 407549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11768.867827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11769.535173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11775.387480] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11785.263958] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11791.701462] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11800.066481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11800.636307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11808.500892] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11818.245689] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11821.041694] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11824.826256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11830.755093] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11830.768269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11830.783618] Lustre: Skipped 1 previous similar message [11830.787218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11832.802578] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11832.814142] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11837.049715] Lustre: server umount lustre-OST0000 complete [11843.040786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11843.063849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11843.074273] Lustre: Skipped 3 previous similar messages [11856.351355] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11856.621517] Lustre: server umount lustre-MDT0000 complete [11857.383652] LustreError: 407549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11857.412233] LustreError: 407549:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11861.584278] LustreError: 407528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787745447 with bad export cookie 6278842238131084459 [11861.587407] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11861.606350] LustreError: 407528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11861.961631] Lustre: server umount lustre-MDT0001 complete [11876.838347] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [11880.760612] Key type lgssc unregistered [11881.172711] LNet: 411279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11881.186961] LNetError: 411279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11881.208912] LNet: Removed LNI 192.168.201.137@tcp [11882.357850] Key type .llcrypt unregistered [11882.366072] Key type ._llcrypt unregistered [11911.805796] Key type ._llcrypt registered [11911.807421] Key type .llcrypt registered [11912.065464] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [11912.549407] Key type .llcrypt unregistered [11912.554668] Key type ._llcrypt unregistered [11938.763319] Key type ._llcrypt registered [11938.768648] Key type .llcrypt registered [11938.934097] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [11940.480296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11940.577123] alg: No test for adler32 (adler32-zlib) [11942.033859] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [11942.355450] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [11944.129325] Key type lgssc registered [11945.816777] Lustre: Echo OBD driver; http://www.lustre.org/ [11956.459672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11956.530752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11958.263476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11964.280859] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11968.489593] LustreError: 413501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11968.525985] LustreError: 413501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11973.621090] LustreError: 413500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11975.130947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11975.595022] LustreError: 413501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11975.615523] LustreError: 413501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11975.638642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11980.771746] LustreError: 413500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11980.788198] LustreError: 413500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11983.285688] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11983.877945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11988.966904] LustreError: 414505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11989.004338] LustreError: 414505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11992.471957] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11999.203888] LustreError: 413501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11999.278884] LustreError: 413501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12000.893776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12001.414031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12006.463647] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12019.687602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12019.712080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12019.732324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12021.731361] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12021.758285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12021.811114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12024.191126] Lustre: server umount lustre-OST0000 complete [12031.973355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12031.993458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12032.027320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12032.040502] Lustre: Skipped 2 previous similar messages [12036.244847] Lustre: server umount lustre-MDT0000 complete [12037.605842] LustreError: 413500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12037.615661] LustreError: 413500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12041.760606] LustreError: 413480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787745627 with bad export cookie 6680271591097220635 [12041.768669] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12041.770021] LustreError: 413480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12042.391459] Lustre: server umount lustre-MDT0001 complete [12057.127439] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [12060.348851] Key type lgssc unregistered [12060.760184] LNet: 416654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12060.771647] LNetError: 416654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12060.785588] LNet: Removed LNI 192.168.201.137@tcp [12062.074065] Key type .llcrypt unregistered [12062.076232] Key type ._llcrypt unregistered [12090.705576] Key type ._llcrypt registered [12090.710090] Key type .llcrypt registered [12090.836586] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [12092.554310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12092.596967] alg: No test for adler32 (adler32-zlib) [12093.862531] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [12094.209631] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [12095.959233] Key type lgssc registered [12097.517793] Lustre: Echo OBD driver; http://www.lustre.org/ [12106.303371] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12106.320770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12121.825171] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12126.947587] LustreError: 418058:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.137@tcp: failed processing log, type 4: rc = -110 [12153.711094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12160.146550] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12168.870764] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12175.806892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12176.698452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12176.776750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12181.999162] LustreError: 418083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12182.021248] LustreError: 418083:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12182.556533] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12187.124807] LustreError: 418084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12188.143933] LustreError: 419095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12188.191245] LustreError: 419095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12193.251337] LustreError: 419095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12193.275671] LustreError: 419095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12193.895643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12194.522519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12201.586179] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12212.243461] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12219.283961] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12228.069785] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12228.082043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12228.092969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12230.626678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12230.631850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12230.666091] Lustre: Skipped 1 previous similar message [12233.999921] Lustre: server umount lustre-OST0000 complete [12240.355848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12240.368531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12240.383503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12245.989614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12246.002581] Lustre: Skipped 2 previous similar messages [12250.092416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12250.108520] Lustre: Skipped 1 previous similar message [12254.177384] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12254.542706] Lustre: server umount lustre-MDT0000 complete [12255.200542] LustreError: 419101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12255.220988] LustreError: 419101:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12259.621539] LustreError: 420587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787745845 with bad export cookie 6803408180620672764 [12259.629795] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12259.640127] LustreError: 420587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12260.312202] Lustre: server umount lustre-MDT0001 complete [12274.631961] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [12278.135721] Key type lgssc unregistered [12278.517660] LNet: 421635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12278.525960] LNetError: 421635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12278.569218] LNet: Removed LNI 192.168.201.137@tcp [12279.694675] Key type .llcrypt unregistered [12279.696406] Key type ._llcrypt unregistered [12309.735527] Key type ._llcrypt registered [12309.738148] Key type .llcrypt registered [12309.858969] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [12310.998574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12311.073031] alg: No test for adler32 (adler32-zlib) [12312.263082] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [12312.506591] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [12314.191208] Key type lgssc registered [12315.711995] Lustre: Echo OBD driver; http://www.lustre.org/ [12325.177126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12325.223889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12326.895184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12331.884845] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12336.116589] LustreError: 423054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12336.147165] LustreError: 423054:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12341.216874] LustreError: 423055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12343.156720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12343.913646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12349.948804] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12357.669234] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12366.307629] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12375.415948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12376.413431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12385.315382] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12395.744452] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12396.999863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12402.158539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12402.163545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12402.169135] Lustre: Skipped 2 previous similar messages [12402.170634] Lustre: Skipped 1 previous similar message [12403.707056] LustreError: 423050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12403.762933] Lustre: server umount lustre-MDT0000 complete [12406.785616] LustreError: 423033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.37@tcp arrived at 1787745992 with bad export cookie 4072406016924868905 [12406.808611] LustreError: 423033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12407.270676] LustreError: 423055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12407.298468] LustreError: 423055:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12409.479389] LustreError: 423034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787745995 with bad export cookie 4072406016924868079 [12409.484170] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12409.488188] LustreError: 423034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12412.394711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12412.395426] LustreError: 423051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12412.412375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12412.412388] Lustre: Skipped 1 previous similar message [12412.441989] LustreError: 423051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12414.434877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12415.757753] Lustre: server umount lustre-OST0000 complete [12424.678923] LustreError: 423049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12424.745108] LustreError: 423049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12426.519470] Lustre: server umount lustre-MDT0001 complete [12445.853623] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [12450.148979] Key type lgssc unregistered [12450.578597] LNet: 426767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12450.584842] LNetError: 426767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12450.623530] LNet: Removed LNI 192.168.201.137@tcp [12451.892472] Key type .llcrypt unregistered [12451.901287] Key type ._llcrypt unregistered [12482.520403] Key type ._llcrypt registered [12482.530751] Key type .llcrypt registered [12482.754100] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [12486.117082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12486.202909] alg: No test for adler32 (adler32-zlib) [12487.480346] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [12487.897406] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [12489.751300] Key type lgssc registered [12492.113392] Lustre: Echo OBD driver; http://www.lustre.org/ [12505.482048] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12505.495523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12507.083749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12514.795193] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12517.350482] LustreError: 428190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12517.379689] LustreError: 428190:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12522.471580] LustreError: 428189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12525.829665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12526.245844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12531.620348] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12540.535269] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12548.570593] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12556.228601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12556.794411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12565.579970] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12575.340542] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12577.284967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:193) [12587.496562] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12587.510182] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [12587.537657] Lustre: Skipped 2 previous similar messages [12592.595450] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [12597.712591] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [12597.720438] Lustre: Skipped 2 previous similar messages [12599.263199] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12599.711529] Lustre: server umount lustre-MDT0001 complete [12602.851992] LustreError: 428185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12602.863558] LustreError: 428185:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12607.968193] LustreError: 428190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12607.985378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12607.985499] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12608.031261] Lustre: Skipped 1 previous similar message [12608.050550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12608.057647] Lustre: Skipped 2 previous similar messages [12610.206821] Lustre: server umount lustre-OST0000 complete [12613.107691] LustreError: 428189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12615.079287] Lustre: server umount lustre-MDT0000 complete [12636.795882] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [12641.309795] Key type lgssc unregistered [12641.781067] LNet: 431905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12641.790456] LNetError: 431905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12641.826270] LNet: Removed LNI 192.168.201.137@tcp [12643.204710] Key type .llcrypt unregistered [12643.209186] Key type ._llcrypt unregistered [12677.160609] Key type ._llcrypt registered [12677.165820] Key type .llcrypt registered [12677.279733] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [12696.403268] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [12697.675985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12697.864368] alg: No test for adler32 (adler32-zlib) [12699.240337] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [12699.519274] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [12701.176479] Key type lgssc registered [12702.212266] Lustre: Echo OBD driver; http://www.lustre.org/ [12752.519754] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [12773.758744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12773.810679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12775.167332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12775.201916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12775.312740] Lustre: lustre-MDT0000: new disk, initializing [12775.486361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12775.521549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12781.283960] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12802.123741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12802.683329] Lustre: lustre-OST0000: new disk, initializing [12802.686055] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12802.689970] Lustre: Skipped 1 previous similar message [12802.695115] Lustre: 436949:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12802.807631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12808.325859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12808.340874] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12808.391477] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12811.468380] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12823.306928] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12843.036456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12843.169446] Lustre: 435926:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12843.220551] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12843.277308] Lustre: lustre-MDT0001: new disk, initializing [12843.374852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12843.412051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12843.425359] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12849.686267] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12862.647876] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12862.990212] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12870.937666] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12871.298187] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12878.837248] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [12881.891937] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12881.895647] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12881.911314] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12881.940357] Lustre: Skipped 1 previous similar message [12881.962407] Lustre: Skipped 1 previous similar message [12883.733787] Lustre: server umount lustre-MDT0001 complete [12889.049408] LustreError: 435934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12889.065249] LustreError: 435934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12892.136575] LustreError: 435937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12892.164337] LustreError: 435937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12897.274454] LustreError: 435938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12897.311504] LustreError: 435938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12902.371200] LustreError: 435937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12902.412588] LustreError: 435937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12905.440615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12905.444845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12905.496292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12907.494091] LustreError: 435938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12910.875704] Lustre: server umount lustre-OST0000 complete [12923.142254] Lustre: server umount lustre-MDT0000 complete [12942.080275] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [12945.278851] Key type lgssc unregistered [12945.749648] LNet: 440703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12945.767906] LNetError: 440703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12945.807240] LNet: Removed LNI 192.168.201.137@tcp [12946.806749] Key type .llcrypt unregistered [12946.808220] Key type ._llcrypt unregistered [12999.006985] Key type ._llcrypt registered [12999.011364] Key type .llcrypt registered [12999.161545] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [12999.598767] Key type .llcrypt unregistered [12999.602244] Key type ._llcrypt unregistered [13021.630778] Key type ._llcrypt registered [13021.633955] Key type .llcrypt registered [13021.813939] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [13023.552288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13023.686405] alg: No test for adler32 (adler32-zlib) [13024.974313] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [13025.412942] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [13027.143202] Key type lgssc registered [13029.398572] Lustre: Echo OBD driver; http://www.lustre.org/ [13062.228812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13062.271789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13063.566341] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13063.606686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13063.705164] Lustre: lustre-MDT0000: new disk, initializing [13063.792300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13063.818762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13069.335666] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [13082.221316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13082.385454] Lustre: 444088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13082.448950] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13082.453571] Lustre: Skipped 1 previous similar message [13082.543567] Lustre: lustre-MDT0001: new disk, initializing [13082.656025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13082.717419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13082.746753] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13088.703566] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [13100.795083] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13107.361510] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13117.608730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13118.397506] Lustre: lustre-OST0000: new disk, initializing [13118.400355] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13118.409328] Lustre: 446060:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13118.571948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13126.116228] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [13126.836385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13126.849490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13127.110257] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13139.918313] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13152.740907] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13152.751225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13152.777956] Lustre: Skipped 2 previous similar messages [13152.812944] Lustre: Skipped 2 previous similar messages [13158.111820] Lustre: server umount lustre-MDT0000 complete [13162.868495] LustreError: 446059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787746748 with bad export cookie 423745027496004642 [13162.871422] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13162.875588] LustreError: 446059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13162.976426] LustreError: 444098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13162.976943] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13162.987364] LustreError: 444098:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13162.999780] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13163.110726] Lustre: server umount lustre-MDT0001 complete [13170.382923] Lustre: server umount lustre-OST0000 complete [13185.005713] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 08:19:29 (1787746769) [13194.056793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13194.795083] LustreError: 448090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13194.902424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13200.357657] LustreError: 448091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13200.557147] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [13204.458792] LustreError: 448090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13209.568844] LustreError: 448091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13210.691528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13211.271941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13216.816486] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [13227.071059] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13236.641680] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13257.876927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13258.336747] Lustre: Found index 0 for lustre-OST0000, updating log [13258.613787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13264.759653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13266.390980] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [13274.760242] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13282.021591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13282.039344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13287.391797] Lustre: server umount lustre-OST0000 complete [13304.804338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13304.818518] Lustre: Skipped 1 previous similar message [13304.828108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13304.836877] Lustre: Skipped 3 previous similar messages [13309.920995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13309.929602] Lustre: Skipped 1 previous similar message [13310.642089] Lustre: server umount lustre-MDT0000 complete [13310.949298] LustreError: 448830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13310.978674] LustreError: 448830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13313.743838] LustreError: 448072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787746899 with bad export cookie 423745027496006035 [13313.750631] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13313.758134] LustreError: 448072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13314.032096] Lustre: server umount lustre-MDT0001 complete [13323.864213] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 08:21:48 (1787746908) [13326.559430] LNet: 1 local NIs in recovery (showing 1): 192.168.201.137@tcp [13957.019905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13957.468691] LustreError: 454505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13957.486991] LustreError: 454505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13957.627030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13962.738741] LustreError: 454506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13963.739053] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [13967.845305] LustreError: 454505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13973.621638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13974.054536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13979.205205] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [13988.231727] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13995.937722] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14016.975169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14017.340419] Lustre: Found index 0 for lustre-OST0000, updating log [14017.696538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14018.805168] LNet: 1 local NIs in recovery (showing 1): 192.168.201.137@tcp [14024.637536] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [14035.397293] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14038.898276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14038.912870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14052.319218] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14052.508774] Lustre: server umount lustre-OST0000 complete [14058.220615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14058.236435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14058.251580] Lustre: Skipped 1 previous similar message [14058.279378] Lustre: Skipped 1 previous similar message [14060.339617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14060.350319] Lustre: Skipped 3 previous similar messages [14063.629367] Lustre: server umount lustre-MDT0000 complete [14064.421654] LustreError: 454506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14064.464262] LustreError: 454506:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [14067.749612] LustreError: 455241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787747653 with bad export cookie 423745027496007106 [14067.752444] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14067.776374] LustreError: 455241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14068.672651] Lustre: server umount lustre-MDT0001 complete [14078.175204] LNet: 1 local NIs in recovery (showing 1): 192.168.201.137@tcp [14102.364922] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [14114.448885] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [14158.812299] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 08:35:42 (1787747742) [14181.086383] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [14185.028513] Key type lgssc unregistered [14185.505500] LNet: 461135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14185.517648] LNetError: 461135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14279.716697] LNet: Removed LNI 192.168.201.137@tcp [14280.828755] Key type .llcrypt unregistered [14280.835867] Key type ._llcrypt unregistered [14299.513579] Key type ._llcrypt registered [14299.516366] Key type .llcrypt registered [14299.613749] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [14300.744622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14300.839708] alg: No test for adler32 (adler32-zlib) [14301.985264] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [14302.313733] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [14303.991245] Key type lgssc registered [14305.397573] Lustre: Echo OBD driver; http://www.lustre.org/ [14310.023806] LNet: Added LNI 192.168.201.137@tcp100 [8/256/0/180] [14311.029695] LNet: Added LNI 192.168.201.137@tcp101 [8/256/0/180] [14311.995306] LNet: Added LNI 192.168.201.137@tcp102 [8/256/0/180] [14312.861172] LNet: Added LNI 192.168.201.137@tcp103 [8/256/0/180] [14313.835382] LNet: Added LNI 192.168.201.137@tcp104 [8/256/0/180] [14314.820140] LNet: Added LNI 192.168.201.137@tcp105 [8/256/0/180] [14315.708679] LNet: Added LNI 192.168.201.137@tcp106 [8/256/0/180] [14316.561059] LNet: Added LNI 192.168.201.137@tcp107 [8/256/0/180] [14317.427702] LNet: Added LNI 192.168.201.137@tcp108 [8/256/0/180] [14318.486992] LNet: Added LNI 192.168.201.137@tcp109 [8/256/0/180] [14319.599734] LNet: Added LNI 192.168.201.137@tcp110 [8/256/0/180] [14320.711320] LNet: Added LNI 192.168.201.137@tcp111 [8/256/0/180] [14321.604579] LNet: Added LNI 192.168.201.137@tcp112 [8/256/0/180] [14322.586991] LNet: Added LNI 192.168.201.137@tcp113 [8/256/0/180] [14323.578898] LNet: Added LNI 192.168.201.137@tcp114 [8/256/0/180] [14324.692916] LNet: Added LNI 192.168.201.137@tcp115 [8/256/0/180] [14325.759130] LNet: Added LNI 192.168.201.137@tcp116 [8/256/0/180] [14326.916660] LNet: Added LNI 192.168.201.137@tcp117 [8/256/0/180] [14328.083322] LNet: Added LNI 192.168.201.137@tcp118 [8/256/0/180] [14328.816128] LNet: Added LNI 192.168.201.137@tcp119 [8/256/0/180] [14329.732531] LNet: Added LNI 192.168.201.137@tcp120 [8/256/0/180] [14330.552642] LNet: Added LNI 192.168.201.137@tcp121 [8/256/0/180] [14331.214774] LNet: Added LNI 192.168.201.137@tcp122 [8/256/0/180] [14331.998942] LNet: Added LNI 192.168.201.137@tcp123 [8/256/0/180] [14332.639985] LNet: Added LNI 192.168.201.137@tcp124 [8/256/0/180] [14333.577370] LNet: Added LNI 192.168.201.137@tcp125 [8/256/0/180] [14334.308632] LNet: Added LNI 192.168.201.137@tcp126 [8/256/0/180] [14335.279604] LNet: Added LNI 192.168.201.137@tcp127 [8/256/0/180] [14336.171419] LNet: Added LNI 192.168.201.137@tcp128 [8/256/0/180] [14337.107744] LNet: Added LNI 192.168.201.137@tcp129 [8/256/0/180] [14338.191896] LNet: Added LNI 192.168.201.137@tcp130 [8/256/0/180] [14339.355994] LNet: Added LNI 192.168.201.137@tcp131 [8/256/0/180] [14340.308421] LNet: Added LNI 192.168.201.137@tcp132 [8/256/0/180] [14341.301934] LNet: Added LNI 192.168.201.137@tcp133 [8/256/0/180] [14342.315905] LNet: Added LNI 192.168.201.137@tcp134 [8/256/0/180] [14343.310842] LNet: Added LNI 192.168.201.137@tcp135 [8/256/0/180] [14350.065112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14350.088678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14351.187950] Lustre: 464165:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.137@tcp: can't add NID 192.168.201.137@tcp130: rc = -75 [14351.279372] LNetError: 461621:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.137 [14351.300349] LNetError: Protocol error connecting to 192.168.201.137@tcp121 at host 192.168.201.137:988: is it running a compatible version of Lustre? [14351.310021] LNetError: Skipped 3 previous similar messages [14351.373481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14351.402537] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14351.478789] Lustre: lustre-MDT0000: new disk, initializing [14351.520065] LNet: 6 local NIs in recovery (showing 6): 192.168.201.137@tcp124, 192.168.201.137@tcp126, 192.168.201.137@tcp121, 192.168.201.137@tcp125, 192.168.201.137@tcp127, 192.168.201.137@tcp129 [14351.562227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14351.590136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14352.352769] LNetError: 461623:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.137 [14352.376573] LNetError: 461623:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [14352.393576] LNetError: Protocol error connecting to 192.168.201.137@tcp108 at host 192.168.201.137:988: is it running a compatible version of Lustre? [14352.405612] LNetError: Skipped 7 previous similar messages [14354.593092] LNetError: 461623:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.137 [14354.600263] LNetError: 461623:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [14354.617525] LNetError: Protocol error connecting to 192.168.201.137@tcp126 at host 192.168.201.137:988: is it running a compatible version of Lustre? [14354.634027] LNetError: Skipped 7 previous similar messages [14356.182259] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [14358.689406] LNetError: 464188:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.137 [14358.696482] LNetError: 464188:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14358.710556] LNetError: Protocol error connecting to 192.168.201.137@tcp126 at host 192.168.201.137:988: is it running a compatible version of Lustre? [14366.885405] LNetError: 464185:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.137 [14366.910602] LNetError: Protocol error connecting to 192.168.201.137@tcp126 at host 192.168.201.137:988: is it running a compatible version of Lustre? [14368.015884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14368.118478] Lustre: 464181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14368.170840] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14368.174369] Lustre: Skipped 1 previous similar message [14368.223835] Lustre: lustre-MDT0001: new disk, initializing [14368.268949] Lustre: 464984:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.137@tcp: can't add NID 192.168.201.137@tcp130: rc = -75 [14368.274676] Lustre: 464984:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14368.394546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14368.446142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14368.467073] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14373.374508] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [14376.486836] Lustre: 465354:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.137@tcp: can't add NID 192.168.201.137@tcp130: rc = -75 [14376.492918] Lustre: 465354:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14381.153299] LNetError: 461624:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.137 [14381.173898] LNetError: Protocol error connecting to 192.168.201.137@tcp123 at host 192.168.201.137:988: is it running a compatible version of Lustre? [14381.184339] LNetError: Skipped 1 previous similar message [14385.331589] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14391.222435] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14397.544688] LNetError: 461624:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.137 [14397.557699] LNetError: 461624:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [14397.571056] LNetError: Protocol error connecting to 192.168.201.137@tcp108 at host 192.168.201.137:988: is it running a compatible version of Lustre? [14397.589804] LNetError: Skipped 10 previous similar messages [14400.925279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14401.315765] Lustre: lustre-OST0000: new disk, initializing [14401.323502] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14401.331713] Lustre: 466158:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14401.497905] Lustre: 466163:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.137@tcp: can't add NID 192.168.201.137@tcp130: rc = -75 [14401.506183] Lustre: 466163:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14401.580312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14409.627458] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [14410.804818] Lustre: 466563:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.137@tcp: can't add NID 192.168.201.137@tcp130: rc = -75 [14410.813339] Lustre: 466563:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14410.877635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14410.893136] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14411.028365] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14411.935519] LNet: 1 local NIs in recovery (showing 1): 192.168.201.137@tcp126 [14421.650895] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14431.328305] LNetError: 464192:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.137 [14431.336232] LNetError: 464192:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 22 previous similar messages [14431.356634] LNetError: Protocol error connecting to 192.168.201.137@tcp120 at host 192.168.201.137:988: is it running a compatible version of Lustre? [14431.390446] LNetError: Skipped 22 previous similar messages [14431.715794] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14431.731507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14431.738228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14431.757291] Lustre: Skipped 1 previous similar message [14431.775238] Lustre: Skipped 1 previous similar message [14436.838236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14436.844493] Lustre: Skipped 1 previous similar message [14441.971701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14441.992518] Lustre: Skipped 1 previous similar message [14444.002596] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14444.857499] Lustre: server umount lustre-OST0000 complete [14450.152147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14450.169533] LustreError: Skipped 1 previous similar message [14450.175610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14450.218168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14452.195815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14455.931200] Lustre: server umount lustre-MDT0000 complete [14459.101708] LustreError: 464174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787748045 with bad export cookie 12842697266997171910 [14459.105803] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14459.112334] LustreError: 464174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14459.425965] Lustre: server umount lustre-MDT0001 complete [14471.087959] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [14474.295830] Key type lgssc unregistered [14474.642283] LNet: 468077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14474.646867] LNetError: 468077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14474.658080] LNet: Removed LNI 192.168.201.137@tcp [14475.679143] LNet: Removed LNI 192.168.201.137@tcp100 [14475.692619] LNetError: 468077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14475.706814] LNetError: 468077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14475.715211] LNet: Removed LNI 192.168.201.137@tcp101 [14476.767302] LNet: Removed LNI 192.168.201.137@tcp102 [14476.771477] LNetError: 468077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14476.775868] LNetError: 468077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14476.781462] LNet: Removed LNI 192.168.201.137@tcp103 [14477.791386] LNet: Removed LNI 192.168.201.137@tcp104 [14478.815193] LNet: Removed LNI 192.168.201.137@tcp105 [14478.817173] LNetError: 468077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14478.830333] LNetError: 468077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14478.854372] LNet: Removed LNI 192.168.201.137@tcp106 [14479.903190] LNet: Removed LNI 192.168.201.137@tcp107 [14480.927190] LNet: Removed LNI 192.168.201.137@tcp108 [14481.951162] LNet: Removed LNI 192.168.201.137@tcp109 [14482.975432] LNet: Removed LNI 192.168.201.137@tcp110 [14482.987973] LNetError: 468077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14483.004723] LNetError: 468077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14483.020820] LNet: Removed LNI 192.168.201.137@tcp111 [14484.064542] LNet: Removed LNI 192.168.201.137@tcp112 [14485.152678] LNet: Removed LNI 192.168.201.137@tcp113 [14486.239316] LNet: Removed LNI 192.168.201.137@tcp114 [14487.263374] LNet: Removed LNI 192.168.201.137@tcp115 [14488.289473] LNet: Removed LNI 192.168.201.137@tcp116 [14489.311771] LNet: Removed LNI 192.168.201.137@tcp117 [14490.335167] LNet: Removed LNI 192.168.201.137@tcp118 [14491.424275] LNet: Removed LNI 192.168.201.137@tcp119 [14491.426697] LNetError: 468077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14491.439667] LNetError: 468077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14491.466704] LNet: Removed LNI 192.168.201.137@tcp120 [14492.511367] LNet: Removed LNI 192.168.201.137@tcp121 [14493.535197] LNet: Removed LNI 192.168.201.137@tcp122 [14494.559156] LNet: Removed LNI 192.168.201.137@tcp123 [14495.584059] LNet: Removed LNI 192.168.201.137@tcp124 [14496.607481] LNet: Removed LNI 192.168.201.137@tcp125 [14496.626289] LNet: Removed LNI 192.168.201.137@tcp126 [14497.703409] LNet: Removed LNI 192.168.201.137@tcp127 [14498.783664] LNet: Removed LNI 192.168.201.137@tcp128 [14499.871228] LNet: Removed LNI 192.168.201.137@tcp129 [14499.873439] LNet: Removed LNI 192.168.201.137@tcp130 [14499.885728] LNet: Removed LNI 192.168.201.137@tcp131 [14499.901470] LNet: Removed LNI 192.168.201.137@tcp132 [14499.906705] LNet: Removed LNI 192.168.201.137@tcp133 [14499.914796] LNet: Removed LNI 192.168.201.137@tcp134 [14499.940631] LNet: Removed LNI 192.168.201.137@tcp135 [14500.577693] Key type .llcrypt unregistered [14500.578695] Key type ._llcrypt unregistered [14526.253766] Key type ._llcrypt registered [14526.257854] Key type .llcrypt registered [14526.363097] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [14545.493758] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [14549.097240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14549.147889] alg: No test for adler32 (adler32-zlib) [14550.596684] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [14550.949880] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [14552.719179] Key type lgssc registered [14555.480636] Lustre: Echo OBD driver; http://www.lustre.org/ [14597.863385] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 08:43:02 (1787748182) [14620.958185] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [14625.962886] Key type lgssc unregistered [14626.430586] LNet: 472173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14626.445731] LNetError: 472173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14626.468525] LNet: Removed LNI 192.168.201.137@tcp [14628.127442] Key type .llcrypt unregistered [14628.130106] Key type ._llcrypt unregistered [14651.952955] Key type ._llcrypt registered [14651.954967] Key type .llcrypt registered [14652.125090] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [14653.967596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14654.388093] alg: No test for adler32 (adler32-zlib) [14655.630311] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [14655.925809] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [14657.648355] Key type lgssc registered [14659.242337] Lustre: Echo OBD driver; http://www.lustre.org/ [14669.617368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14669.665291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14671.136159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14671.196466] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14671.286442] Lustre: lustre-MDT0000: new disk, initializing [14671.474046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14671.499192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14676.968460] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [14688.415788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14688.562274] Lustre: 473392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14688.617184] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14688.621541] Lustre: Skipped 1 previous similar message [14688.685773] Lustre: lustre-MDT0001: new disk, initializing [14688.770706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14688.818165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14688.833481] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14694.597530] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [14705.203779] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14711.362422] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14722.023637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14722.590677] Lustre: lustre-OST0000: new disk, initializing [14722.610791] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14722.628784] Lustre: 475365:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14722.841765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14730.793143] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [14731.417569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14731.428676] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14731.612420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14741.656552] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14747.457955] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing lnet_if_list [14753.530955] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14753.556689] LNet: Added LNI 192.168.201.137@tcp10 [8/256/0/180] [14760.599521] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14760.641746] LNet: Added LNI 192.168.201.137@tcp11 [8/256/0/180] [14767.244895] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14767.286541] LNet: Added LNI 192.168.201.137@tcp12 [8/256/0/180] [14772.240938] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14772.281900] LNet: Added LNI 192.168.201.137@tcp13 [8/256/0/180] [14777.763993] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14777.786220] LNet: Added LNI 192.168.201.137@tcp14 [8/256/0/180] [14783.556809] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14783.591337] LNet: Added LNI 192.168.201.137@tcp15 [8/256/0/180] [14789.936074] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14789.957751] LNet: Added LNI 192.168.201.137@tcp16 [8/256/0/180] [14796.309386] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14796.331320] LNet: Added LNI 192.168.201.137@tcp17 [8/256/0/180] [14802.773759] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14802.795692] LNet: Added LNI 192.168.201.137@tcp18 [8/256/0/180] [14808.644780] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14808.685687] LNet: Added LNI 192.168.201.137@tcp19 [8/256/0/180] [14815.777523] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14815.829660] LNet: Added LNI 192.168.201.137@tcp20 [8/256/0/180] [14823.103170] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14823.124946] LNet: Added LNI 192.168.201.137@tcp21 [8/256/0/180] [14829.988632] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14830.017034] LNet: Added LNI 192.168.201.137@tcp22 [8/256/0/180] [14836.641539] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14836.662583] LNet: Added LNI 192.168.201.137@tcp23 [8/256/0/180] [14844.239272] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14844.283020] LNet: Added LNI 192.168.201.137@tcp24 [8/256/0/180] [14851.071784] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14851.111427] LNet: Added LNI 192.168.201.137@tcp25 [8/256/0/180] [14858.434175] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14858.459679] LNet: Added LNI 192.168.201.137@tcp26 [8/256/0/180] [14866.746803] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14866.770180] LNet: Added LNI 192.168.201.137@tcp27 [8/256/0/180] [14874.642858] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14874.690167] LNet: Added LNI 192.168.201.137@tcp28 [8/256/0/180] [14882.976128] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14883.005267] LNet: Added LNI 192.168.201.137@tcp29 [8/256/0/180] [14891.480533] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14891.561374] LNet: Added LNI 192.168.201.137@tcp30 [8/256/0/180] [14899.876945] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14899.903481] LNet: Added LNI 192.168.201.137@tcp31 [8/256/0/180] [14906.982715] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14907.020758] LNet: Added LNI 192.168.201.137@tcp32 [8/256/0/180] [14917.594709] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14917.644893] LNet: Added LNI 192.168.201.137@tcp33 [8/256/0/180] [14928.438351] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14928.492131] LNet: Added LNI 192.168.201.137@tcp34 [8/256/0/180] [14935.817556] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14935.835887] LNet: Added LNI 192.168.201.137@tcp35 [8/256/0/180] [14945.530226] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14945.552761] LNet: Added LNI 192.168.201.137@tcp36 [8/256/0/180] [14954.770513] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14954.824949] LNet: Added LNI 192.168.201.137@tcp37 [8/256/0/180] [14964.251362] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14964.282856] LNet: Added LNI 192.168.201.137@tcp38 [8/256/0/180] [14973.024402] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14973.054599] LNet: Added LNI 192.168.201.137@tcp39 [8/256/0/180] [14981.975409] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14982.002868] LNet: Added LNI 192.168.201.137@tcp40 [8/256/0/180] [14991.674630] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14991.696515] LNet: Added LNI 192.168.201.137@tcp41 [8/256/0/180] [15000.096056] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15000.117581] LNet: Added LNI 192.168.201.137@tcp42 [8/256/0/180] [15010.612424] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15010.633559] LNet: Added LNI 192.168.201.137@tcp43 [8/256/0/180] [15020.486894] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15020.522918] LNet: Added LNI 192.168.201.137@tcp44 [8/256/0/180] [15030.183527] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15030.220791] LNet: Added LNI 192.168.201.137@tcp45 [8/256/0/180] [15039.980808] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15040.001329] LNet: Added LNI 192.168.201.137@tcp46 [8/256/0/180] [15050.786716] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15050.831641] LNet: Added LNI 192.168.201.137@tcp47 [8/256/0/180] [15061.690978] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15061.754892] LNet: Added LNI 192.168.201.137@tcp48 [8/256/0/180] [15073.423640] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15073.452029] LNet: Added LNI 192.168.201.137@tcp49 [8/256/0/180] [15084.050989] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15084.100534] LNet: Added LNI 192.168.201.137@tcp50 [8/256/0/180] [15103.457589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15103.460419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15103.490899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15105.508964] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15105.509989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15105.532207] Lustre: Skipped 1 previous similar message [15105.570351] Lustre: Skipped 2 previous similar messages [15109.705212] Lustre: server umount lustre-MDT0000 complete [15114.165051] LustreError: 473383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787748700 with bad export cookie 12209743791071341126 [15114.178461] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15114.186236] LustreError: 473383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15114.900299] Lustre: server umount lustre-MDT0001 complete [15131.935244] Lustre: 472663:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787748701/real 1787748701] req@00000000c238aebe x1874589899751552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787748717 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15131.968331] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15135.211723] Lustre: 472660:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787748705/real 1787748705] req@000000009144b894 x1874589899751680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787748721 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15136.287396] Lustre: 472660:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787748706/real 1787748706] req@000000009d1a7643 x1874589899751808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787748722 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15137.593498] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [15140.259933] Lustre: 472662:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787748710/real 1787748710] req@000000002aa4aa5b x1874589899752064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787748726 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15146.463153] Lustre: 472662:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787748716/real 1787748716] req@0000000058d84413 x1874589899752192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787748732 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15146.818134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15160.133482] LustreError: 475357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15160.152954] LustreError: 475357:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15160.259664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15161.250920] LustreError: 475358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15161.287994] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15166.235217] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [15166.442100] LustreError: 475357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15171.557114] LustreError: 475358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15176.377041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15176.673983] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [15176.690761] Lustre: Skipped 1 previous similar message [15176.919828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15182.312769] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15182.346305] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [15191.475473] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15197.820404] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15221.729052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15221.742057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15221.770256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15223.293187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15226.145505] Lustre: server umount lustre-OST0000 complete [15233.509367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15233.515804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15233.530943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15233.531732] Lustre: Skipped 2 previous similar messages [15233.547092] Lustre: Skipped 3 previous similar messages [15237.882892] Lustre: server umount lustre-MDT0000 complete [15238.625816] LustreError: 499615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15238.641544] LustreError: 499615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15242.510674] LustreError: 473383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787748828 with bad export cookie 12209743791071343800 [15242.530820] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15243.037302] Lustre: server umount lustre-MDT0001 complete [15256.732610] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [15260.073444] Key type lgssc unregistered [15260.615760] LNet: 502422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15260.624229] LNetError: 502422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15260.633492] LNet: Removed LNI 192.168.201.137@tcp [15260.642640] LNet: Removed LNI 192.168.201.137@tcp10 [15260.647755] LNet: Removed LNI 192.168.201.137@tcp11 [15260.659127] LNet: Removed LNI 192.168.201.137@tcp12 [15260.666527] LNet: Removed LNI 192.168.201.137@tcp13 [15260.675960] LNet: Removed LNI 192.168.201.137@tcp14 [15260.681961] LNet: Removed LNI 192.168.201.137@tcp15 [15260.687991] LNet: Removed LNI 192.168.201.137@tcp16 [15260.702778] LNet: Removed LNI 192.168.201.137@tcp17 [15260.711456] LNet: Removed LNI 192.168.201.137@tcp18 [15260.720583] LNet: Removed LNI 192.168.201.137@tcp19 [15260.727822] LNet: Removed LNI 192.168.201.137@tcp20 [15260.733238] LNet: Removed LNI 192.168.201.137@tcp21 [15260.738193] LNet: Removed LNI 192.168.201.137@tcp22 [15260.748253] LNet: Removed LNI 192.168.201.137@tcp23 [15260.752685] LNet: Removed LNI 192.168.201.137@tcp24 [15260.756912] LNet: Removed LNI 192.168.201.137@tcp25 [15260.763718] LNet: Removed LNI 192.168.201.137@tcp26 [15260.771388] LNet: Removed LNI 192.168.201.137@tcp27 [15260.778139] LNet: Removed LNI 192.168.201.137@tcp28 [15260.784862] LNet: Removed LNI 192.168.201.137@tcp29 [15260.789404] LNet: Removed LNI 192.168.201.137@tcp30 [15260.793309] LNet: Removed LNI 192.168.201.137@tcp31 [15260.795682] LNet: Removed LNI 192.168.201.137@tcp32 [15260.798684] LNet: Removed LNI 192.168.201.137@tcp33 [15260.805574] LNet: Removed LNI 192.168.201.137@tcp34 [15260.808820] LNet: Removed LNI 192.168.201.137@tcp35 [15260.812418] LNet: Removed LNI 192.168.201.137@tcp36 [15260.815384] LNet: Removed LNI 192.168.201.137@tcp37 [15260.818611] LNet: Removed LNI 192.168.201.137@tcp38 [15260.821753] LNet: Removed LNI 192.168.201.137@tcp39 [15260.824703] LNet: Removed LNI 192.168.201.137@tcp40 [15260.830706] LNet: Removed LNI 192.168.201.137@tcp41 [15260.835134] LNet: Removed LNI 192.168.201.137@tcp42 [15260.839878] LNet: Removed LNI 192.168.201.137@tcp43 [15260.843244] LNet: Removed LNI 192.168.201.137@tcp44 [15260.847284] LNet: Removed LNI 192.168.201.137@tcp45 [15260.853073] LNet: Removed LNI 192.168.201.137@tcp46 [15260.857798] LNet: Removed LNI 192.168.201.137@tcp47 [15260.866881] LNet: Removed LNI 192.168.201.137@tcp48 [15260.878921] LNet: Removed LNI 192.168.201.137@tcp49 [15260.888873] LNet: Removed LNI 192.168.201.137@tcp50 [15261.935747] Key type .llcrypt unregistered [15261.938483] Key type ._llcrypt unregistered [15285.787317] Key type ._llcrypt registered [15285.791313] Key type .llcrypt registered [15285.982404] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [15303.280518] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [15304.753057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15305.078265] alg: No test for adler32 (adler32-zlib) [15306.350722] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [15306.738656] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [15308.439189] Key type lgssc registered [15310.131930] Lustre: Echo OBD driver; http://www.lustre.org/ [15351.863473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15351.950534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15353.477359] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15353.581188] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15353.757075] Lustre: lustre-MDT0000: new disk, initializing [15353.885564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15353.949672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15359.973890] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [15373.390673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15373.596476] Lustre: 505621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15373.737815] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15373.752486] Lustre: Skipped 1 previous similar message [15373.889640] Lustre: lustre-MDT0001: new disk, initializing [15374.053391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15374.104272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15374.119075] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15380.875796] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [15392.742222] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15400.875435] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15410.692577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15410.960130] Lustre: *** cfs_fail_loc=90f, val=0*** [15411.061355] Lustre: lustre-OST0000: new disk, initializing [15411.067932] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15411.080853] Lustre: 507637:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15411.227833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15418.015276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15418.032632] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15418.221708] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15420.237251] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [15431.246507] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15443.936461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15443.940331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15443.944703] LustreError: Skipped 1 previous similar message [15443.958088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15443.966019] Lustre: Skipped 1 previous similar message [15443.998314] Lustre: Skipped 1 previous similar message [15445.652657] Lustre: server umount lustre-OST0000 complete [15451.104136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15451.112782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15451.135938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15454.179331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15454.186180] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15456.352782] Lustre: server umount lustre-MDT0000 complete [15460.167866] LustreError: 506415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787749046 with bad export cookie 17456088290966123772 [15460.173625] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15460.180873] LustreError: 506415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15460.751333] Lustre: server umount lustre-MDT0001 complete [15474.712613] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [15478.072814] Key type lgssc unregistered [15478.335832] LNet: 509551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15478.341510] LNetError: 509551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15478.361483] LNet: Removed LNI 192.168.201.137@tcp [15479.318944] Key type .llcrypt unregistered [15479.322861] Key type ._llcrypt unregistered [15502.487985] Key type ._llcrypt registered [15502.490474] Key type .llcrypt registered [15502.585868] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [15504.052675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15504.096403] alg: No test for adler32 (adler32-zlib) [15505.260453] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [15505.537168] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [15507.287188] Key type lgssc registered [15509.041989] Lustre: Echo OBD driver; http://www.lustre.org/ [15518.262747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15518.280830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15519.923457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15525.351314] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [15529.451559] LustreError: 510822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15529.475891] LustreError: 510822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15534.561814] LustreError: 510823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15536.022604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15536.431021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15541.937700] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [15551.083195] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15557.289333] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15565.161992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15565.640903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15572.085808] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [15580.713224] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15601.666605] loop0: detected capacity change from 0 to 204800000 [15601.984445] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15602.009445] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15602.032373] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15609.199756] loop0: detected capacity change from 0 to 204800000 [15609.550759] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15609.578169] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15609.598574] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15615.574582] loop0: detected capacity change from 0 to 204800000 [15622.286082] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15622.507637] Lustre: fs15246-OST0000: new disk, initializing [15622.519289] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15622.527422] Lustre: 514438:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15622.590417] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15630.597579] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [15641.805404] loop1: detected capacity change from 0 to 204800000 [15648.409593] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15648.592371] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15648.672991] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15648.762505] Lustre: fs15246-MDT0000: new disk, initializing [15648.944344] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15648.970787] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15649.107608] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15650.144209] Lustre: 515560:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000bc3735e4 x1874590790897536/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787749252 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15655.202622] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15655.215532] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [15655.226549] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15655.298187] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15683.566048] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15683.578697] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15688.347822] Lustre: server umount fs15246-MDT0000 complete [15688.682919] LustreError: 510819:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15693.809585] LustreError: 510817:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15698.889857] Lustre: server umount fs15246-OST0000 complete [15712.226223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15712.247765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15712.268613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15714.272379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15714.286084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15719.393505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15719.403728] Lustre: Skipped 2 previous similar messages [15724.519932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15726.047773] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15726.337974] Lustre: server umount lustre-OST0000 complete [15731.171019] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15731.189248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15734.754587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15734.766284] Lustre: Skipped 3 previous similar messages [15737.836876] Lustre: server umount lustre-MDT0000 complete [15739.878971] LustreError: 510823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15739.905883] LustreError: 510823:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15742.300440] LustreError: 510803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787749328 with bad export cookie 1974785984027753113 [15742.304639] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15742.315545] LustreError: 510803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15742.815962] Lustre: server umount lustre-MDT0001 complete [15757.976832] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [15761.395618] Key type lgssc unregistered [15761.624698] LNet: 518673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15761.628743] LNetError: 518673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15761.650909] LNet: Removed LNI 192.168.201.137@tcp [15762.857624] Key type .llcrypt unregistered [15762.859292] Key type ._llcrypt unregistered [15788.129260] Key type ._llcrypt registered [15788.131834] Key type .llcrypt registered [15788.310791] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [15790.321809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15790.464689] alg: No test for adler32 (adler32-zlib) [15791.793788] Lustre: Lustre: Build Version: 2.17.57_80_g6de652b [15792.148670] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [15793.911199] Key type lgssc registered [15795.509926] Lustre: Echo OBD driver; http://www.lustre.org/ [15844.722413] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [15857.451500] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [15899.988986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15900.067702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15901.515233] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15901.587263] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15901.755139] Lustre: lustre-MDT0000: new disk, initializing [15901.852493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15901.892883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15906.918110] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [15920.380729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15920.630630] Lustre: 523146:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15920.706988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15920.713446] Lustre: Skipped 1 previous similar message [15920.909250] Lustre: lustre-MDT0001: new disk, initializing [15921.145113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15921.244377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15921.264861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15928.806175] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [15941.304837] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15948.939273] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15958.341550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15958.741938] Lustre: lustre-OST0000: new disk, initializing [15958.750417] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15958.763411] Lustre: 525116:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15958.912886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15967.362858] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [15968.950722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15968.984506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15969.376139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15979.428105] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15985.156641] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15985.354861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15991.027617] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15991.309362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15994.850776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15994.870266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15994.899715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15994.909128] Lustre: Skipped 1 previous similar message [15999.840564] Lustre: server umount lustre-OST0000 complete [16005.096452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16005.104751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16005.118777] Lustre: Skipped 2 previous similar messages [16005.131259] Lustre: Skipped 1 previous similar message [16010.211544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16010.220633] Lustre: Skipped 1 previous similar message [16011.146966] Lustre: server umount lustre-MDT0000 complete [16015.954271] LustreError: 523140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787749601 with bad export cookie 15193813206870041869 [16015.968994] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16015.971270] LustreError: 523140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16016.488380] Lustre: server umount lustre-MDT0001 complete [16045.745673] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [16058.698416] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [16097.927617] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15835 sec ========= 09:08:02 (1787749682) [16100.137962] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:08:04 (1787749684) === [16104.681495] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:08:09 (1787749689) === [16154.315652] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [16158.396244] Key type lgssc unregistered [16158.830828] LNet: 531770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16158.846555] LNetError: 531770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16158.886347] LNet: Removed LNI 192.168.201.137@tcp [16160.231540] Key type .llcrypt unregistered [16160.233499] Key type ._llcrypt unregistered