[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 476817836 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003172] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.008291] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011127] LSM: Security Framework initializing [ 0.012094] Yama: becoming mindful. [ 0.013036] SELinux: Initializing. [ 0.014063] *** VALIDATE selinux *** [ 0.022723] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027427] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028137] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030101] *** VALIDATE tmpfs *** [ 0.031431] *** VALIDATE proc *** [ 0.033065] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035247] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036201] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038026] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.042797] debug: unmapping init [mem 0xffffffffb8e59000-0xffffffffb8e60fff] [ 0.044178] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045609] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046020] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048015] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050010] ... max period: 00007fffffffffff [ 0.051014] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053278] rcu: Hierarchical SRCU implementation. [ 0.055439] smp: Bringing up secondary CPUs ... [ 0.056613] x86: Booting SMP configuration: [ 0.057025] .... node #0, CPUs: #1 #2 #3 [ 0.060493] smp: Brought up 1 node, 4 CPUs [ 0.062011] smpboot: Max logical packages: 1 [ 0.063011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136284] node 0 deferred pages initialised in 70ms [ 0.141198] devtmpfs: initialized [ 0.142252] x86/mm: Memory block size: 128MB [ 0.144696] gcov: version magic: 0x41383552 [ 0.146021] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150085] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152322] pinctrl core: initialized pinctrl subsystem [ 0.154183] [ 0.154741] ************************************************************* [ 0.157011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159009] ** ** [ 0.160061] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162010] ** ** [ 0.164011] ** This means that this kernel is built to expose internal ** [ 0.166010] ** IOMMU data structures, which may compromise security on ** [ 0.168011] ** your system. ** [ 0.170064] ** ** [ 0.172011] ** If you see this message and you are not debugging the ** [ 0.174010] ** kernel, report this immediately to your vendor! ** [ 0.176009] ** ** [ 0.178010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.180017] ************************************************************* [ 0.182582] NET: Registered protocol family 16 [ 0.184468] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.187069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.190058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.193057] cpuidle: using governor menu [ 0.194858] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.197370] PCI: Using configuration type 1 for base access [ 0.199112] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.208055] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.209023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.212062] cryptd: max_cpu_qlen set to 1000 [ 0.214213] ACPI: Added _OSI(Module Device) [ 0.215013] ACPI: Added _OSI(Processor Device) [ 0.216009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.217011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.222203] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.227371] ACPI: Interpreter enabled [ 0.229072] ACPI: PM: (supports S0 S3 S4 S5) [ 0.230009] ACPI: Using IOAPIC for interrupt routing [ 0.232106] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.234374] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.245244] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.247038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.249018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.253072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.258333] acpiphp: Slot [2] registered [ 0.259134] acpiphp: Slot [5] registered [ 0.261110] acpiphp: Slot [6] registered [ 0.262119] acpiphp: Slot [7] registered [ 0.264104] acpiphp: Slot [8] registered [ 0.265081] acpiphp: Slot [9] registered [ 0.266101] acpiphp: Slot [10] registered [ 0.268192] acpiphp: Slot [3] registered [ 0.269088] acpiphp: Slot [4] registered [ 0.271105] acpiphp: Slot [11] registered [ 0.273105] acpiphp: Slot [12] registered [ 0.274098] acpiphp: Slot [13] registered [ 0.276105] acpiphp: Slot [14] registered [ 0.278120] acpiphp: Slot [15] registered [ 0.279108] acpiphp: Slot [16] registered [ 0.281097] acpiphp: Slot [17] registered [ 0.283101] acpiphp: Slot [18] registered [ 0.284163] acpiphp: Slot [19] registered [ 0.286072] acpiphp: Slot [20] registered [ 0.288113] acpiphp: Slot [21] registered [ 0.290085] acpiphp: Slot [22] registered [ 0.292141] acpiphp: Slot [23] registered [ 0.293102] acpiphp: Slot [24] registered [ 0.294073] acpiphp: Slot [25] registered [ 0.296096] acpiphp: Slot [26] registered [ 0.297106] acpiphp: Slot [27] registered [ 0.298084] acpiphp: Slot [28] registered [ 0.301149] acpiphp: Slot [29] registered [ 0.302081] acpiphp: Slot [30] registered [ 0.303109] acpiphp: Slot [31] registered [ 0.304054] PCI host bridge to bus 0000:00 [ 0.305016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.307020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.309024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.311017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.314022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.316021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.318148] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.319786] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.323000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.331527] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.335063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.338018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.340017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.342020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.346443] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.348982] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.350052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.353834] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.358013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.371011] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.376014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.381000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.388014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.397015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.421016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.434073] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.448015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.456014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.481025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.494063] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.504023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.515022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.544021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.553696] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.568030] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.574017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.589015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.599254] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.607025] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.615025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.635026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.653466] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.662022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.673022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.692024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.704915] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.707426] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.710441] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.713458] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.715287] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.722053] iommu: Default domain type: Passthrough [ 0.724678] SCSI subsystem initialized [ 0.726246] ACPI: bus type USB registered [ 0.727161] usbcore: registered new interface driver usbfs [ 0.729137] usbcore: registered new interface driver hub [ 0.731113] usbcore: registered new device driver usb [ 0.733220] pps_core: LinuxPPS API ver. 1 registered [ 0.735017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.738084] PTP clock support registered [ 0.741075] EDAC MC: Ver: 3.0.0 [ 0.743190] PCI: Using ACPI for IRQ routing [ 0.744597] NetLabel: Initializing [ 0.746023] NetLabel: domain hash size = 128 [ 0.747017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.749122] NetLabel: unlabeled traffic allowed by default [ 0.752112] vgaarb: loaded [ 0.753284] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.755026] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.760343] clocksource: Switched to clocksource kvm-clock [ 0.873128] VFS: Disk quotas dquot_6.6.0 [ 0.874983] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.877669] *** VALIDATE ramfs *** [ 0.879265] *** VALIDATE hugetlbfs *** [ 0.880848] pnp: PnP ACPI init [ 0.883630] pnp: PnP ACPI: found 6 devices [ 0.903615] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.907426] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.910077] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.912591] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.915506] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.918041] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.921135] NET: Registered protocol family 2 [ 0.923589] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.930664] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.934851] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.939791] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.943442] TCP: Hash tables configured (established 65536 bind 65536) [ 0.946652] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.950162] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.954482] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.957483] NET: Registered protocol family 1 [ 0.960918] RPC: Registered named UNIX socket transport module. [ 0.963232] RPC: Registered udp transport module. [ 0.965159] RPC: Registered tcp transport module. [ 0.967029] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.969533] NET: Registered protocol family 44 [ 0.971351] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.973760] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.976197] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.990669] pci 0000:00:01.0: quirk_isa_dma_hangs+0x0/0x20 took 14131 usecs [ 0.996114] PCI: CLS 0 bytes, default 64 [ 1.001967] Unpacking initramfs... [ 2.449826] debug: unmapping init [mem 0xffff91acbcc54000-0xffff91acbffbffff] [ 2.454398] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.457019] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.460040] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.988102] Initialise system trusted keyrings [ 2.991222] Key type blacklist registered [ 2.993443] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.002768] zbud: loaded [ 3.005732] *** VALIDATE nfs *** [ 3.007119] *** VALIDATE nfs4 *** [ 3.008907] pstore: using deflate compression [ 3.013896] Platform Keyring initialized [ 3.128451] NET: Registered protocol family 38 [ 3.130675] Key type asymmetric registered [ 3.132697] Asymmetric key parser 'x509' registered [ 3.135045] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.138847] io scheduler mq-deadline registered [ 3.140698] io scheduler kyber registered [ 3.142622] io scheduler bfq registered [ 3.144893] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.148219] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.151503] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.155137] ACPI: Power Button [PWRF] [ 3.161709] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.168969] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.185377] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.196369] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.230634] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.259697] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.289469] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.295838] Non-volatile memory driver v1.3 [ 3.297617] Linux agpgart interface v0.103 [ 3.334922] virtio_blk virtio1: [vda] 146944 512-byte logical blocks (75.2 MB/71.8 MiB) [ 3.337893] vda: detected capacity change from 0 to 75235328 [ 3.352829] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.355816] vdb: detected capacity change from 0 to 1073741824 [ 3.374153] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.377626] vdc: detected capacity change from 0 to 2621440000 [ 3.393630] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.396740] vdd: detected capacity change from 0 to 2621440000 [ 3.422119] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.424743] vde: detected capacity change from 0 to 4294967296 [ 3.443078] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.446022] vdf: detected capacity change from 0 to 4294967296 [ 3.459946] libphy: Fixed MDIO Bus: probed [ 3.466609] usbcore: registered new interface driver usbserial_generic [ 3.469558] usbserial: USB Serial support registered for generic [ 3.472379] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.477109] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.479146] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.482245] mousedev: PS/2 mouse device common for all mice [ 3.485677] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.487925] rtc_cmos 00:05: RTC can wake from S4 [ 3.493615] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.494063] rtc_cmos 00:05: registered as rtc0 [ 3.498523] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.499849] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.501498] intel_pstate: CPU model not supported [ 3.508536] hid: raw HID events driver (C) Jiri Kosina [ 3.510606] usbcore: registered new interface driver usbhid [ 3.512435] usbhid: USB HID core driver [ 3.513881] drop_monitor: Initializing network drop monitor service [ 3.516332] Initializing XFRM netlink socket [ 3.518417] NET: Registered protocol family 10 [ 3.521820] Segment Routing with IPv6 [ 3.523249] NET: Registered protocol family 17 [ 3.525316] mpls_gso: MPLS GSO support [ 3.531661] RAS: Correctable Errors collector initialized. [ 3.533668] AVX version of gcm_enc/dec engaged. [ 3.535514] AES CTR mode by8 optimization enabled [ 3.618398] sched_clock: Marking stable (3618370628, 0)->(4504282116, -885911488) [ 3.622288] registered taskstats version 1 [ 3.625105] Loading compiled-in X.509 certificates [ 3.628274] zswap: loaded using pool lzo/zbud [ 3.653083] Key type big_key registered [ 3.666890] Key type encrypted registered [ 3.668500] ima: No TPM chip found, activating TPM-bypass! [ 3.670356] ima: Allocated hash algorithm: sha1 [ 3.672047] ima: No architecture policies found [ 3.673722] evm: Initialising EVM extended attributes: [ 3.675700] evm: security.selinux [ 3.677200] evm: security.ima [ 3.678124] evm: security.capability [ 3.679334] evm: HMAC attrs: 0x1 [ 3.681786] rtc_cmos 00:05: setting system clock to 2026-08-27 14:35:37 UTC (1787841337) [ 3.689134] debug: unmapping init [mem 0xffffffffb9e03000-0xffffffffb9ffffff] [ 3.693632] debug: unmapping init [mem 0xffffffffb8b82000-0xffffffffb8e58fff] [ 3.702140] Write protecting the kernel read-only data: 28672k [ 3.707671] debug: unmapping init [mem 0xffffffffb7203000-0xffffffffb73fffff] [ 3.711296] debug: unmapping init [mem 0xffffffffb7b14000-0xffffffffb7bfffff] [ 3.748974] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.758632] systemd[1]: Detected virtualization kvm. [ 3.760815] systemd[1]: Detected architecture x86-64. [ 3.762827] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.791103] systemd[1]: No hostname configured. [ 3.793457] systemd[1]: Set hostname to . [ 3.795796] random: systemd: uninitialized urandom read (16 bytes read) [ 3.798548] systemd[1]: Initializing machine ID from random generator. [ 3.946560] random: systemd: uninitialized urandom read (16 bytes read) [ 3.949286] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.953914] random: systemd: uninitialized urandom read (16 bytes read) [ 3.956753] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.962418] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.605306] device-mapper: uevent: version 1.0.3 [ 4.607867] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.298753] random: fast init done [ 5.315974] virtio_net virtio0 ens2: renamed from eth0 [ 5.331691] scsi host0: ata_piix [ 5.360322] scsi host1: ata_piix [ 5.420682] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.423720] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.104069] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.188417] random: crng init done [ 10.190220] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.563743] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.665730] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.914454] SELinux: Disabled at runtime. [ 11.979547] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.988644] systemd[1]: Detected virtualization kvm. [ 11.990644] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.489444] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.493204] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.498358] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.502441] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.506239] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.514098] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.521381] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Jour[ 12.688278] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS nal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.938607] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.286787] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.311528] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.435288] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.458250] EDAC sbridge: Ver: 1.1.2 [ 15.004893] Key type dns_resolver registered [ 15.307410] NFS: Registering the id_resolver key type [ 15.309578] Key type id_resolver registered [ 15.311826] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg605-server login: [ 59.653397] libcfs: loading out-of-tree module taints kernel. [ 59.824059] Key type ._llcrypt registered [ 59.830239] Key type .llcrypt registered [ 59.966685] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 76.482903] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 78.254501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 78.291958] alg: No test for adler32 (adler32-zlib) [ 79.996706] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 81.115434] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 82.984772] Key type lgssc registered [ 85.019307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 102.362097] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 142.421193] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 155.026099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 155.078442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 156.388616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 156.419807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 156.510629] Lustre: lustre-MDT0000: new disk, initializing [ 156.599280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 156.631492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 161.587655] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 175.524068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 175.662720] Lustre: 6499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 175.729197] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 175.735500] Lustre: Skipped 1 previous similar message [ 175.906701] Lustre: lustre-MDT0001: new disk, initializing [ 175.934809] hrtimer: interrupt took 4800556 ns [ 176.017682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 176.072158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 176.089966] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 181.064493] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 186.177795] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 197.248666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.655896] Lustre: lustre-OST0000: new disk, initializing [ 197.659620] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 197.669515] Lustre: 8436:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 197.734309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 201.142932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 201.149492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 201.274653] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 206.403342] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.777559] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 221.962716] Lustre: lustre-OST0001: new disk, initializing [ 221.966591] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 221.977309] Lustre: 9509:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 222.044482] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 228.602310] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.493653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 231.503200] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 231.562579] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 240.926854] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 247.457468] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 253.666429] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing check_logdir /tmp/testlogs/ [ 258.924518] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing yml_node [ 262.730647] Lustre: DEBUG MARKER: Client: 2.17.57.95 [ 265.243405] Lustre: DEBUG MARKER: MDS: 2.17.57.95 [ 267.631501] Lustre: DEBUG MARKER: OSS: 2.17.57.95 [ 269.420336] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 10:40:01 EDT 2026 [ 286.031197] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 287.683988] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 304.096339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 304.102439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.118162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 308.193180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 308.195084] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.014486] Lustre: server umount lustre-MDT0000 complete [ 313.719860] LustreError: 9508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841647 with bad export cookie 15101733477143789939 [ 313.722264] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 313.729201] LustreError: 9508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 318.432586] LustreError: 6505:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 318.438942] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.442378] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 318.442389] Lustre: Skipped 3 previous similar messages [ 318.444868] LustreError: 6505:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 318.450170] Lustre: Skipped 3 previous similar messages [ 323.553161] LustreError: 6504:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 323.554870] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 323.573830] LustreError: 6504:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 323.590374] Lustre: Skipped 2 previous similar messages [ 328.159783] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 328.529640] Lustre: server umount lustre-MDT0001 complete [ 332.058550] Lustre: server umount lustre-OST0000 complete [ 335.373752] Lustre: server umount lustre-OST0001 complete [ 341.673489] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 351.104747] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 386.275234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 386.575137] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 386.617825] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 386.690352] Lustre: lustre-MDT0000: new disk, initializing [ 386.764730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 386.787062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 391.596494] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 402.012624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 402.121402] Lustre: 15064:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 402.170485] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 402.173671] Lustre: Skipped 1 previous similar message [ 402.245613] Lustre: lustre-MDT0001: new disk, initializing [ 402.328249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 402.366392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 402.382712] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 406.476574] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 416.242425] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 422.423937] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 429.030039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 429.426712] Lustre: lustre-OST0000: new disk, initializing [ 429.431985] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 429.446129] Lustre: 17032:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 429.558134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 430.874597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 430.889945] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 431.001937] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 436.010227] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 446.287195] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 451.341792] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 451.543424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 456.057256] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 456.305674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 461.794615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 461.804848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 461.815307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 461.821633] Lustre: Skipped 1 previous similar message [ 464.597947] Lustre: server umount lustre-OST0000 complete [ 468.960959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 468.970614] LustreError: Skipped 1 previous similar message [ 468.976982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 472.032791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 472.037678] Lustre: Skipped 2 previous similar messages [ 474.983141] Lustre: server umount lustre-MDT0000 complete [ 477.160275] LustreError: 15076:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 477.177243] LustreError: 15076:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 479.052748] LustreError: 15859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841812 with bad export cookie 15101733477143791472 [ 479.054125] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 479.067734] LustreError: 15859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 479.509429] Lustre: server umount lustre-MDT0001 complete [ 484.005491] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:43:36 (1787841816) [ 486.391702] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 488.451653] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:43:40 (1787841820) [ 495.014594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.591585] LustreError: 19442:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.676691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 500.713769] LustreError: 19443:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.840351] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 510.627424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 510.947261] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 516.416582] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 524.973541] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 530.779217] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 539.006535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 547.165760] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 555.690290] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 566.248651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 566.249600] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 566.273975] Lustre: Skipped 1 previous similar message [ 566.276805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 566.318898] Lustre: Skipped 1 previous similar message [ 568.505880] Lustre: server umount lustre-OST0000 complete [ 572.384646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 572.391117] LustreError: Skipped 1 previous similar message [ 578.552873] Lustre: server umount lustre-MDT0000 complete [ 581.602385] LustreError: 19439:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 581.620266] LustreError: 19439:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 581.907303] LustreError: 19423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841915 with bad export cookie 15101733477143792354 [ 581.913718] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 582.153271] Lustre: server umount lustre-MDT0001 complete [ 590.836806] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 593.190275] Key type lgssc unregistered [ 593.433187] LNet: 22977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.439201] LNetError: 22977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 593.452916] LNet: Removed LNI 192.168.206.105@tcp [ 594.362287] Key type .llcrypt unregistered [ 594.366725] Key type ._llcrypt unregistered [ 616.524568] Key type ._llcrypt registered [ 616.533071] Key type .llcrypt registered [ 616.655400] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 618.258502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 618.280380] alg: No test for adler32 (adler32-zlib) [ 619.278444] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 619.518325] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 621.271165] Key type lgssc registered [ 622.457550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 630.999545] loop: module loaded [ 631.128730] loop0: detected capacity change from 0 to 204800000 [ 631.477658] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 631.494491] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 631.836192] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 637.966504] loop0: detected capacity change from 0 to 204800000 [ 638.191060] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 638.208944] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 638.238097] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 643.268073] loop0: detected capacity change from 0 to 204800000 [ 649.655587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 649.693341] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.031843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 651.093539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 651.226839] Lustre: lustre-MDT0000: new disk, initializing [ 651.344061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 651.381463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 656.629968] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 667.126318] loop1: detected capacity change from 0 to 204800000 [ 673.698093] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 674.125936] Lustre: lustre-OST07c6: new disk, initializing [ 674.129601] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 674.133973] Lustre: Skipped 1 previous similar message [ 674.138424] Lustre: 26165:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 674.260978] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 681.272912] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 681.656917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 681.666198] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 681.727258] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 686.539737] Lustre: 25148:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 686.546954] Lustre: 25148:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 686.558211] Lustre: 25148:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 686.569785] Lustre: 25148:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 686.583138] Lustre: 25148:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 686.590652] Lustre: 25148:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 709.239493] Lustre: 25148:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 709.250693] Lustre: 25148:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 709.260343] Lustre: 25148:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 709.273578] Lustre: 25148:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 709.282861] Lustre: 25148:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 709.289648] Lustre: 25148:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 714.213340] Lustre: 26682:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 714.243839] Lustre: 26682:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 714.262088] Lustre: 26682:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 714.280861] Lustre: 26682:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 714.305958] Lustre: 26682:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 714.314813] Lustre: 26682:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 714.323437] Lustre: 26682:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 714.338559] Lustre: 26682:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 714.345464] Lustre: 26682:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 714.356032] Lustre: 26682:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 714.362191] Lustre: 26682:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 714.369338] Lustre: 26682:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 717.558940] Lustre: 25148:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 717.580747] Lustre: 25148:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 717.590754] Lustre: 25148:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 717.600138] Lustre: 25148:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 717.609415] Lustre: 25148:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 717.615757] Lustre: 25148:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 717.622721] Lustre: 25148:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 717.631584] Lustre: 25148:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 717.636401] Lustre: 25148:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 717.640955] Lustre: 25148:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 717.645085] Lustre: 25148:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 717.649424] Lustre: 25148:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.388263] Lustre: 26682:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 722.405101] Lustre: 26682:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 722.412427] Lustre: 26682:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 722.425028] Lustre: 26682:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.437165] Lustre: 26682:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 722.446620] Lustre: 26682:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.458606] Lustre: 26682:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 722.468712] Lustre: 26682:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.481948] Lustre: 26682:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 722.489851] Lustre: 26682:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.499187] Lustre: 26682:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 722.509242] Lustre: 26682:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 733.729617] Lustre: 26682:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 733.751567] Lustre: 26682:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 733.763428] Lustre: 26682:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 733.769516] Lustre: 26682:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 733.779111] Lustre: 26682:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 733.787581] Lustre: 26682:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 733.807506] Lustre: 26682:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 733.819313] Lustre: 26682:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 733.825938] Lustre: 26682:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 733.830941] Lustre: 26682:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 733.836298] Lustre: 26682:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 733.840820] Lustre: 26682:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 783.850548] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 783.866778] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 786.731810] Lustre: server umount lustre-OST07c6 complete [ 795.675640] Lustre: server umount lustre-MDT0000 complete [ 810.393927] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 812.703860] Key type lgssc unregistered [ 812.940252] LNet: 28728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 812.949944] LNetError: 28728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 812.978658] LNet: Removed LNI 192.168.206.105@tcp [ 813.693236] Key type .llcrypt unregistered [ 813.696827] Key type ._llcrypt unregistered [ 830.714679] Key type ._llcrypt registered [ 830.716969] Key type .llcrypt registered [ 830.792687] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 831.892109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 831.930705] alg: No test for adler32 (adler32-zlib) [ 832.918989] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 833.120994] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 834.791312] Key type lgssc registered [ 835.733758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 842.731717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 842.751819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 844.207388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 848.250715] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 853.477374] LustreError: 29999:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 853.497377] LustreError: 29999:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 856.140262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 856.473689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 861.106949] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 867.135859] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 872.736662] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 879.197539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 879.467279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 885.098597] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 888.715987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 892.613103] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 907.243614] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 907.250714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 907.265247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 909.286606] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 909.287789] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 909.288533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 912.497421] Lustre: server umount lustre-OST0000 complete [ 917.985085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 917.997260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 918.017818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 918.023732] Lustre: Skipped 1 previous similar message [ 924.641574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 924.653530] Lustre: Skipped 3 previous similar messages [ 929.762379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 929.767912] Lustre: Skipped 1 previous similar message [ 930.272897] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 930.433875] Lustre: server umount lustre-MDT0000 complete [ 933.793208] LustreError: 29980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842267 with bad export cookie 9614534083595982898 [ 933.807204] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 933.811528] LustreError: 29980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 934.129244] Lustre: server umount lustre-MDT0001 complete [ 943.326616] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 945.845623] Key type lgssc unregistered [ 946.109806] LNet: 33740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.127035] LNetError: 33740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.145198] LNet: Removed LNI 192.168.206.105@tcp [ 946.903200] Key type .llcrypt unregistered [ 946.906252] Key type ._llcrypt unregistered [ 965.505209] Key type ._llcrypt registered [ 965.507881] Key type .llcrypt registered [ 965.599685] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 966.346530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 966.490116] alg: No test for adler32 (adler32-zlib) [ 967.541306] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 967.790609] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 969.463145] Key type lgssc registered [ 970.550595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 977.221684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 977.242073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 978.597933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 983.187779] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 989.155169] LustreError: 35010:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 989.170559] LustreError: 35010:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 991.706647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 992.232275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 996.742470] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1003.991855] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1010.121966] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1016.540533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1016.854383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1022.825307] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1024.054525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1029.747681] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1039.343385] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1039.362734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1039.371925] Lustre: Skipped 1 previous similar message [ 1044.448680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1044.459856] Lustre: Skipped 1 previous similar message [ 1045.003931] Lustre: server umount lustre-OST0000 complete [ 1049.571168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1049.574613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1049.579547] Lustre: Skipped 2 previous similar messages [ 1049.588025] Lustre: Skipped 1 previous similar message [ 1054.689378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1054.695761] Lustre: Skipped 1 previous similar message [ 1054.972902] Lustre: server umount lustre-MDT0000 complete [ 1058.261687] LustreError: 34989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842392 with bad export cookie 3180128634141499069 [ 1058.264159] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1058.270631] LustreError: 34989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1058.599849] Lustre: server umount lustre-MDT0001 complete [ 1069.134806] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 1071.593938] Key type lgssc unregistered [ 1071.895952] LNet: 38546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.907706] LNetError: 38546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.932080] LNet: Removed LNI 192.168.206.105@tcp [ 1072.668357] Key type .llcrypt unregistered [ 1072.672781] Key type ._llcrypt unregistered [ 1091.899395] Key type ._llcrypt registered [ 1091.902304] Key type .llcrypt registered [ 1092.009288] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 1093.272124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1093.320908] alg: No test for adler32 (adler32-zlib) [ 1094.399938] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 1094.739950] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 1096.431302] Key type lgssc registered [ 1097.687083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1107.232787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1107.250623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1108.760511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1114.482433] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1119.205885] LustreError: 39943:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1119.229885] LustreError: 39943:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1123.870137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1124.266790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1129.120765] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1136.619288] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1142.773666] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1151.727114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1152.233253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1154.294807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1160.331286] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1169.346521] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1180.140578] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1180.150092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1180.155722] Lustre: Skipped 1 previous similar message [ 1180.173500] Lustre: Skipped 2 previous similar messages [ 1182.283271] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.5@tcp (stopping) [ 1183.599474] Lustre: server umount lustre-MDT0000 complete [ 1185.257271] LustreError: 39940:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1185.279257] LustreError: 39940:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1187.372773] LustreError: 39938:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1187.430073] LustreError: 39924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842521 with bad export cookie 5403025366267699421 [ 1187.445149] LustreError: 39924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1187.447144] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1190.368168] LustreError: 39939:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1190.368537] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1190.412434] Lustre: Skipped 1 previous similar message [ 1190.417930] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1192.418969] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1193.888619] Lustre: server umount lustre-MDT0001 complete [ 1215.548449] Lustre: server umount lustre-OST0000 complete [ 1236.942633] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 1242.080703] Key type lgssc unregistered [ 1242.351748] LNet: 43811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1242.362676] LNetError: 43811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1242.382507] LNet: Removed LNI 192.168.206.105@tcp [ 1243.275707] Key type .llcrypt unregistered [ 1243.281563] Key type ._llcrypt unregistered [ 1268.477675] Key type ._llcrypt registered [ 1268.481946] Key type .llcrypt registered [ 1268.639559] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 1269.656660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1269.684237] alg: No test for adler32 (adler32-zlib) [ 1270.748337] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 1271.040819] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 1272.831172] Key type lgssc registered [ 1274.046445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1281.447693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1281.468138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1283.034848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1288.040987] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1292.263464] LustreError: 45230:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1292.282151] LustreError: 45230:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1296.574666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1296.908992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1301.794629] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1308.663934] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1314.145123] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1321.795122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1322.185962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1327.294095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1328.869869] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1336.943589] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1347.042773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1347.047620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1347.053631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1347.056272] Lustre: Skipped 1 previous similar message [ 1347.658758] Lustre: server umount lustre-OST0000 complete [ 1366.698322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1366.713350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1366.719629] Lustre: Skipped 1 previous similar message [ 1366.731560] Lustre: Skipped 2 previous similar messages [ 1372.074960] Lustre: server umount lustre-MDT0000 complete [ 1372.136150] LustreError: 45230:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.152387] LustreError: 45230:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1376.099088] LustreError: 45212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842709 with bad export cookie 12882155137196907889 [ 1376.102801] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1376.114856] LustreError: 45212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1382.722514] Lustre: server umount lustre-MDT0001 complete [ 1394.293576] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 1397.074945] Key type lgssc unregistered [ 1397.407793] LNet: 48941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1397.424867] LNetError: 48941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1397.449791] LNet: Removed LNI 192.168.206.105@tcp [ 1398.155265] Key type .llcrypt unregistered [ 1398.158778] Key type ._llcrypt unregistered [ 1423.404676] Key type ._llcrypt registered [ 1423.407940] Key type .llcrypt registered [ 1423.547412] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 1424.669698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1424.800973] alg: No test for adler32 (adler32-zlib) [ 1425.830142] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 1426.131519] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 1427.871321] Key type lgssc registered [ 1429.181906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1437.836117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1437.861238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1439.377485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1443.649711] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1449.957623] LustreError: 50359:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1449.974566] LustreError: 50359:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1452.652427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1453.050678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1457.409523] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1465.303918] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1470.925732] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1477.372147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1477.604328] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1477.699430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1483.271497] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1484.810523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1491.383859] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1498.767684] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1503.041875] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1515.256979] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2711h00m46s [ 1516.877396] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1519.586330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1519.595377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1519.608600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1520.684833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1520.699714] Lustre: Skipped 5 previous similar messages [ 1525.029870] Lustre: server umount lustre-MDT0000 complete [ 1525.816134] LustreError: 52625:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1528.904580] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1531.971396] LustreError: 50354:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1532.011312] LustreError: 50354:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1535.921526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1536.043358] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1536.353660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1536.439368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1537.403244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1541.215397] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1545.258501] Lustre: DEBUG MARKER: Wait for df (36708) ... [ 1547.057296] Lustre: DEBUG MARKER: done [ 1551.847307] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1551.862174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1551.885787] Lustre: Skipped 2 previous similar messages [ 1551.907205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1551.921831] Lustre: Skipped 1 previous similar message [ 1552.868392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1557.642555] Lustre: server umount lustre-OST0000 complete [ 1562.080497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1562.083548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1562.097558] Lustre: Skipped 1 previous similar message [ 1562.115470] Lustre: Skipped 3 previous similar messages [ 1567.589635] Lustre: server umount lustre-MDT0000 complete [ 1569.249264] LustreError: 50358:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1569.262944] LustreError: 50358:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 1571.179702] LustreError: 52987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842904 with bad export cookie 6934224275066972789 [ 1571.192569] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1571.196480] LustreError: 52987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1571.754750] Lustre: server umount lustre-MDT0001 complete [ 1582.003176] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 1584.503196] Key type lgssc unregistered [ 1584.765906] LNet: 55300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1584.780761] LNetError: 55300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1584.795439] LNet: Removed LNI 192.168.206.105@tcp [ 1585.524439] Key type .llcrypt unregistered [ 1585.527578] Key type ._llcrypt unregistered [ 1618.625648] Key type ._llcrypt registered [ 1618.631554] Key type .llcrypt registered [ 1618.798896] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 1620.715813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1620.753675] alg: No test for adler32 (adler32-zlib) [ 1622.043652] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 1622.295971] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 1623.951149] Key type lgssc registered [ 1625.159894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1633.343764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1633.365680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1634.562393] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1634.574196] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1634.749124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1639.482104] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1648.592897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1648.727844] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1648.784412] Lustre: 57488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1649.012496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1654.043475] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1662.557285] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1668.654129] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1676.311027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1676.567238] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1676.765378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1683.082118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1683.172771] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1691.187990] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1698.574383] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2711h03m50s [ 1700.303892] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1706.916091] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1709.195645] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1709.251992] LustreError: 57497:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1729.622917] Lustre: lustre-MDT0000: Client f4fed576-abff-42c4-8d61-e5be6620bc09 (at 192.168.206.5@tcp) reconnecting [ 1749.265565] LustreError: 57497:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1751.892305] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1754.342771] Lustre: DEBUG MARKER: done [ 1764.832991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1764.850163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1764.886990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1765.355221] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1765.369403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1765.389803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1765.399172] Lustre: Skipped 1 previous similar message [ 1770.466187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1770.476087] Lustre: Skipped 1 previous similar message [ 1775.590498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1775.596164] Lustre: Skipped 1 previous similar message [ 1776.095144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1776.246603] Lustre: server umount lustre-OST0000 complete [ 1780.705692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.706558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1780.720840] Lustre: Skipped 1 previous similar message [ 1786.439077] Lustre: server umount lustre-MDT0000 complete [ 1790.647442] LustreError: 59360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843124 with bad export cookie 17750584472326901553 [ 1790.647684] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1790.664224] LustreError: 59360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1791.330475] Lustre: server umount lustre-MDT0001 complete [ 1802.330368] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 1805.141593] Key type lgssc unregistered [ 1805.414377] LNet: 61826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1805.426981] LNetError: 61826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1806.437326] LNet: Removed LNI 192.168.206.105@tcp [ 1807.313813] Key type .llcrypt unregistered [ 1807.316923] Key type ._llcrypt unregistered [ 1845.545298] Key type ._llcrypt registered [ 1845.547078] Key type .llcrypt registered [ 1845.649905] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 1846.472193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1846.578069] alg: No test for adler32 (adler32-zlib) [ 1847.606652] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 1847.809664] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 1849.495160] Key type lgssc registered [ 1850.508177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1853.196483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1853.221574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1854.526812] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1854.547958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1854.673369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1855.440672] Lustre: Failing over lustre-MDT0000 [ 1855.659572] Lustre: server umount lustre-MDT0000 complete [ 1861.777532] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 11:06:34 (1787843194) [ 1867.573752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1867.896691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1872.114588] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1879.847966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1879.982550] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1880.064829] Lustre: 64603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1880.322613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1884.952468] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1892.226405] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1897.921915] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1904.219036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1904.466820] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1904.639332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1910.825798] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1911.952703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1917.734582] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1923.595930] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1930.820901] Lustre: server umount lustre-MDT0000 complete [ 1931.746266] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1931.754376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1932.770926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1932.782762] Lustre: Skipped 1 previous similar message [ 1933.859780] LustreError: 65350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843267 with bad export cookie 6318710068682983999 [ 1933.862346] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1933.869950] LustreError: 65350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1934.350761] Lustre: server umount lustre-MDT0001 complete [ 1937.354362] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1941.368036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1949.485238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1953.247222] Lustre: 63527:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843271/real 1787843271] req@0000000046ef91e7 x1874689451398912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787843287 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1953.280625] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1954.271266] Lustre: 63528:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843272/real 1787843272] req@000000000a5dbd7c x1874689451399040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787843288 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1958.431255] Lustre: 63528:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843276/real 1787843276] req@00000000be9f47ff x1874689451399168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787843292 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1958.743241] LustreError: 66454:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1958.764093] LustreError: 66454:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 1958.887805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1959.003543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1963.453612] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1964.003941] LustreError: 66455:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1964.025396] LustreError: 66455:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1965.024053] Lustre: 63527:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843282/real 1787843282] req@000000005c6879a4 x1874689451399552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787843298 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1965.073435] Lustre: 63527:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1967.018525] LustreError: 68371:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1967.037609] LustreError: 68371:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1969.120994] Lustre: 63528:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843286/real 1787843286] req@000000003aa10ce9 x1874689451399808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787843302 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1971.624362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1972.036448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1973.107853] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1976.310537] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1983.432748] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1988.397418] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1994.840450] Lustre: server umount lustre-MDT0000 complete [ 1997.804404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1997.823763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1997.851230] LustreError: 69178:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1998.694792] LustreError: 64597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843332 with bad export cookie 6318710068682986274 [ 1998.695727] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1998.708405] LustreError: 64597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1998.820512] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1999.104239] Lustre: server umount lustre-MDT0001 complete [ 2005.888465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2014.848247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2023.392156] LustreError: 63525:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000002fddab7 x1874689451426560/t0(0) o250->MGC192.168.206.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2023.771011] LustreError: 66455:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2023.817408] LustreError: 66455:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2023.888773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2023.930913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2028.854095] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2034.146333] LustreError: 66455:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2034.175680] LustreError: 66455:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2038.268286] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2038.275151] Lustre: Skipped 1 previous similar message [ 2038.754478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2043.799708] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2044.407687] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2052.302602] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2058.044913] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2064.587422] Lustre: server umount lustre-MDT0000 complete [ 2064.868624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2064.870141] LustreError: 66455:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2064.880554] Lustre: Skipped 4 previous similar messages [ 2064.888562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2064.894492] LustreError: 66455:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2068.098212] LustreError: 64597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843401 with bad export cookie 6318710068682987653 [ 2068.119637] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2068.413523] Lustre: server umount lustre-MDT0001 complete [ 2074.695631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2077.677987] Lustre: DEBUG MARKER: files compared the same [ 2089.826873] Lustre: server umount lustre-OST0000 complete [ 2104.702910] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 2107.296851] Key type lgssc unregistered [ 2107.583663] LNet: 74333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2107.589961] LNetError: 74333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2108.656971] LNet: Removed LNI 192.168.206.105@tcp [ 2109.456803] Key type .llcrypt unregistered [ 2109.460316] Key type ._llcrypt unregistered [ 2128.266196] Key type ._llcrypt registered [ 2128.268976] Key type .llcrypt registered [ 2128.369936] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 2129.230529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2129.250082] alg: No test for adler32 (adler32-zlib) [ 2130.258647] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 2130.460917] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 2132.103615] Key type lgssc registered [ 2133.061793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2140.134487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2140.156034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2141.564215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2146.086643] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug +malloc all [ 2151.904486] LustreError: 75731:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2151.923255] LustreError: 75731:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2154.615915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2154.972715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2159.418323] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug +malloc all [ 2167.163790] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2172.596934] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2178.680456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2179.011141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2185.332640] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug +malloc all [ 2187.178070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2193.131835] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2200.786657] Lustre: server umount lustre-OST0000 complete [ 2206.175606] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2206.185332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2206.203732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2207.716526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2207.718231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2207.735355] Lustre: Skipped 1 previous similar message [ 2212.833074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2212.838383] Lustre: Skipped 1 previous similar message [ 2213.937835] Lustre: server umount lustre-MDT0000 complete [ 2217.545709] LustreError: 75711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843551 with bad export cookie 15647750069032275267 [ 2217.547950] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2217.559539] LustreError: 75711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2228.347308] Lustre: server umount lustre-MDT0001 complete [ 2237.603572] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 2240.007059] Key type lgssc unregistered [ 2240.242472] LNet: 79269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2240.255893] LNetError: 79269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2240.275130] LNet: Removed LNI 192.168.206.105@tcp [ 2240.962376] Key type .llcrypt unregistered [ 2240.966093] Key type ._llcrypt unregistered [ 2260.389123] Key type ._llcrypt registered [ 2260.395494] Key type .llcrypt registered [ 2260.483917] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 2261.327158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2261.350670] alg: No test for adler32 (adler32-zlib) [ 2262.326895] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 2262.513488] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 2264.176046] Key type lgssc registered [ 2265.026772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2271.288761] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2271.302872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2286.815417] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2291.935301] LustreError: 80637:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.105@tcp: failed processing log, type 4: rc = -110 [ 2318.559157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2324.480674] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2332.446571] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2340.298466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2340.401269] Lustre: *** cfs_fail_loc=706, val=0*** [ 2340.656265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2340.711498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2344.586700] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2345.963079] LustreError: 80663:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2345.987035] LustreError: 80663:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2351.081980] LustreError: 80662:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.575345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2353.030212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2357.268347] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2365.532321] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2371.488824] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2373.607559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2373.620339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2373.643450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2374.628764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2374.630845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2379.796657] Lustre: server umount lustre-OST0000 complete [ 2384.864904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2384.867636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2384.885945] Lustre: Skipped 1 previous similar message [ 2384.889816] Lustre: Skipped 1 previous similar message [ 2389.990763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2390.005273] Lustre: Skipped 2 previous similar messages [ 2390.301652] Lustre: server umount lustre-MDT0000 complete [ 2393.971941] LustreError: 80643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843727 with bad export cookie 10158446260180643565 [ 2393.981444] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2393.983559] LustreError: 80643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2394.374525] Lustre: server umount lustre-MDT0001 complete [ 2406.355689] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 2409.616449] Key type lgssc unregistered [ 2409.909348] LNet: 84259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2409.921942] LNetError: 84259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2409.936243] LNet: Removed LNI 192.168.206.105@tcp [ 2410.881788] Key type .llcrypt unregistered [ 2410.884326] Key type ._llcrypt unregistered [ 2436.191370] Key type ._llcrypt registered [ 2436.194700] Key type .llcrypt registered [ 2436.386710] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 2437.763356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2438.047482] alg: No test for adler32 (adler32-zlib) [ 2439.285641] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 2439.648440] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 2441.351292] Key type lgssc registered [ 2442.931846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2451.400934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2451.441052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2456.934984] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2464.974621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2465.439578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2469.767616] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2470.880919] LustreError: 86364:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2470.904974] LustreError: 86364:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2476.002064] LustreError: 86367:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2476.033106] LustreError: 86367:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2478.160468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2478.365939] LustreError: 86364:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2478.384874] LustreError: 86364:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2478.400678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2483.681716] LustreError: 86363:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2483.710551] LustreError: 86363:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2484.729411] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2488.810564] LustreError: 86367:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2488.839168] LustreError: 86367:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2493.447932] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2493.469263] Lustre: MGS: Client 1af57e07-e3d5-41b9-ab6e-706a35e70a5f (at 0@lo) reconnecting [ 2493.493697] LustreError: 86378:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2493.813946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2493.930371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2499.034809] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2514.407898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2514.412504] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2514.424497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2519.521639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2519.530082] Lustre: Skipped 2 previous similar messages [ 2520.102870] Lustre: server umount lustre-OST0000 complete [ 2524.646334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2524.647659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2524.664029] Lustre: Skipped 1 previous similar message [ 2524.666246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2524.684463] LustreError: Skipped 1 previous similar message [ 2529.761818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2529.779086] Lustre: Skipped 2 previous similar messages [ 2530.347535] Lustre: server umount lustre-MDT0000 complete [ 2533.946271] Lustre: server umount lustre-MDT0001 complete [ 2537.276572] Lustre: server umount MGS complete [ 2547.263676] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 11:17:59 (1787843879) [ 2575.958327] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 2585.361599] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 2621.974334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2626.015505] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2635.437365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2635.749894] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2635.855277] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2636.012784] Lustre: lustre-MDT0001: new disk, initializing [ 2636.077689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2636.112263] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2636.123121] Lustre: 93550:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2636.142384] LustreError: 93550:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2641.189606] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2653.048397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2653.432900] Lustre: lustre-OST0000: new disk, initializing [ 2653.436614] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2653.448887] Lustre: 94318:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2653.547862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2660.747471] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2661.026180] Lustre: 94611:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000a759bd3d x1874690072158080/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787844010 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2671.339944] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2677.988785] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2677.998052] Lustre: MGS: Client 13a03b99-0c91-44de-bc9f-d070036ecbb5 (at 0@lo) reconnecting [ 2678.024505] Lustre: 92820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2678.042633] LustreError: 93536:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2678.054102] LustreError: 93536:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2678.075655] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2678.148323] Lustre: lustre-MDT0000: new disk, initializing [ 2678.258722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2678.279090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2678.314245] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2679.331569] Lustre: 95390:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000048f5dd5 x1874690072164864/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787844029 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2682.352786] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2684.447983] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2688.441275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2688.453911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2688.517821] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2712.036465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2712.040933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2712.053484] Lustre: Skipped 2 previous similar messages [ 2712.064473] Lustre: Skipped 1 previous similar message [ 2717.491834] Lustre: server umount lustre-OST0000 complete [ 2722.273464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2722.280585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2722.284216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2722.284225] Lustre: Skipped 2 previous similar messages [ 2722.313705] Lustre: Skipped 1 previous similar message [ 2727.535265] Lustre: server umount lustre-MDT0000 complete [ 2737.966713] Lustre: server umount lustre-MDT0001 complete [ 2741.838252] Lustre: server umount MGS complete [ 2752.285777] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 11:21:24 (1787844084) [ 2771.035904] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 2773.767861] Key type lgssc unregistered [ 2774.098804] LNet: 97891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2774.104662] LNetError: 97891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2775.141861] LNet: Removed LNI 192.168.206.105@tcp [ 2776.065818] Key type .llcrypt unregistered [ 2776.068148] Key type ._llcrypt unregistered [ 2793.418984] Key type ._llcrypt registered [ 2793.422570] Key type .llcrypt registered [ 2793.525904] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 2794.601077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2794.632894] alg: No test for adler32 (adler32-zlib) [ 2795.689463] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 2795.931446] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 2797.567213] Key type lgssc registered [ 2798.598626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2805.571418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2805.595918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2807.013140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2811.162518] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2816.483936] LustreError: 99112:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2816.498761] LustreError: 99112:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2819.683289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2820.125467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2824.849953] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2832.276701] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2838.284701] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2844.314221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2844.693644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2850.343447] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2851.843779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2857.749903] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2867.170663] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2867.172524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2867.185374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2872.163400] Lustre: server umount lustre-OST0000 complete [ 2876.388805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2876.396349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2876.413544] Lustre: Skipped 1 previous similar message [ 2876.427441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2876.438154] Lustre: Skipped 1 previous similar message [ 2877.408809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2882.539359] LustreError: 99112:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2882.560417] LustreError: 99112:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2882.725161] Lustre: server umount lustre-MDT0000 complete [ 2886.231306] LustreError: 99094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844220 with bad export cookie 3109519885550906406 [ 2886.236989] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2886.240914] LustreError: 99094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2886.560642] Lustre: server umount lustre-MDT0001 complete [ 2897.609397] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 2900.515522] Key type lgssc unregistered [ 2900.795903] LNet: 102650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2900.806480] LNetError: 102650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2900.829493] LNet: Removed LNI 192.168.206.105@tcp [ 2901.693187] Key type .llcrypt unregistered [ 2901.696975] Key type ._llcrypt unregistered [ 2916.465684] Key type ._llcrypt registered [ 2916.471145] Key type .llcrypt registered [ 2916.564746] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 2917.383064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2917.457220] alg: No test for adler32 (adler32-zlib) [ 2918.521085] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 2918.804806] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 2920.463206] Key type lgssc registered [ 2921.580260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2927.327302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2927.330257] LustreError: 103725:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2927.333157] Lustre: Skipped 1 previous similar message [ 2927.340047] LustreError: 103722:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2927.344656] LustreError: 103725:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4992 [ 2927.349066] LustreError: 103725:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2927.359896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.360964] LustreError: 103725:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2928.788569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2935.843919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2936.136514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2940.462722] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2945.256462] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing lsmod [ 2947.977615] LustreError: 104924:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2948.038329] LustreError: 104931:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2948.051481] LustreError: 104924:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4946 [ 2948.060939] LustreError: 104924:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2948.074369] LustreError: 104924:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2948.091728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2948.691725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2953.696602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2953.709504] Lustre: Skipped 1 previous similar message [ 2953.711760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2953.731678] Lustre: Skipped 2 previous similar messages [ 2958.303975] Lustre: server umount lustre-MDT0000 complete [ 2958.819538] LustreError: 103764:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2958.851194] LustreError: 103764:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2960.873414] LustreError: 103763:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2960.915171] LustreError: 103763:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2962.750712] LustreError: 103740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844296 with bad export cookie 11603032856829226641 [ 2962.756415] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2962.787057] LustreError: 103740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2963.006349] LustreError: 104194:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b3d1812d x1874690574405632/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2963.191445] Lustre: server umount lustre-MDT0001 complete [ 2976.326511] Lustre: server umount lustre-OST0000 complete [ 2982.959665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2983.483743] LustreError: 105999:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2983.584730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2987.995420] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2989.032795] LustreError: 106000:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2994.148765] LustreError: 105999:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2995.869300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2996.188726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2999.664887] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3005.124370] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3009.785774] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3016.804229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3017.234363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3023.895291] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3028.523551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3031.255355] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3038.695416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3038.699757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3038.706890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3038.717936] Lustre: Skipped 2 previous similar messages [ 3043.372437] Lustre: server umount lustre-OST0000 complete [ 3047.393291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3047.403554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3047.420268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3047.425117] Lustre: Skipped 1 previous similar message [ 3052.716931] Lustre: server umount lustre-MDT0000 complete [ 3054.048074] LustreError: 106000:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3054.061144] LustreError: 106000:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3056.121928] LustreError: 105980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844389 with bad export cookie 11603032856829227341 [ 3056.124120] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3056.132861] LustreError: 105980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3056.428607] Lustre: server umount lustre-MDT0001 complete [ 3066.568607] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 3068.952568] Key type lgssc unregistered [ 3069.200330] LNet: 109534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3069.221429] LNetError: 109534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3069.245211] LNet: Removed LNI 192.168.206.105@tcp [ 3069.976968] Key type .llcrypt unregistered [ 3069.982193] Key type ._llcrypt unregistered [ 3090.423052] Key type ._llcrypt registered [ 3090.424686] Key type .llcrypt registered [ 3090.512283] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 3091.407368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3091.425710] alg: No test for adler32 (adler32-zlib) [ 3092.427580] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 3092.634238] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 3094.319169] Key type lgssc registered [ 3095.349996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3102.386553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3102.423985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3103.782058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3107.593347] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3113.962504] LustreError: 110929:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3113.974769] LustreError: 110929:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3115.176219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3115.528303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3119.186690] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3125.354166] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3130.018386] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3135.435868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3135.758943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3141.099868] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3148.504961] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3149.048505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3154.107854] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3156.783528] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3161.571645] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3161.580891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3161.591019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3162.024396] Lustre: server umount lustre-OST0000 complete [ 3166.695105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3166.699530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3166.712117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3169.760681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3169.763680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3169.783378] Lustre: Skipped 1 previous similar message [ 3174.880555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3174.887331] Lustre: Skipped 1 previous similar message [ 3175.997842] Lustre: server umount lustre-MDT0000 complete [ 3179.456811] LustreError: 110909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844513 with bad export cookie 11817482634752346492 [ 3179.462533] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3179.469125] LustreError: 110909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3179.760980] Lustre: server umount lustre-MDT0001 complete [ 3191.670262] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 3194.074669] Key type lgssc unregistered [ 3194.302190] LNet: 114568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3194.308445] LNetError: 114568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3194.324965] LNet: Removed LNI 192.168.206.105@tcp [ 3195.079966] Key type .llcrypt unregistered [ 3195.083205] Key type ._llcrypt unregistered [ 3208.898698] Key type ._llcrypt registered [ 3208.900839] Key type .llcrypt registered [ 3208.992167] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 3209.705972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3209.809071] alg: No test for adler32 (adler32-zlib) [ 3210.822178] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 3211.010341] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 3212.647652] Key type lgssc registered [ 3213.913360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3221.336567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3221.358556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3222.783789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3227.237993] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3232.235676] LustreError: 115769:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3232.249963] LustreError: 115769:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3235.172205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3235.460124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3239.492937] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3246.952595] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3252.925990] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3260.428564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3260.812813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3267.033449] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3268.021341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3275.120731] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3288.554460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3288.555842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3288.561217] LustreError: Skipped 1 previous similar message [ 3288.587881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3290.599075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3290.600229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3290.607936] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3295.716762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3295.726657] Lustre: Skipped 2 previous similar messages [ 3300.835291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3300.846572] Lustre: Skipped 1 previous similar message [ 3302.367197] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3302.579707] Lustre: server umount lustre-OST0000 complete [ 3306.981127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3306.989871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3307.002383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3311.074573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3314.662555] Lustre: server umount lustre-MDT0000 complete [ 3316.194221] LustreError: 116834:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3316.208257] LustreError: 116834:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3318.272213] LustreError: 115749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844652 with bad export cookie 17346090860049046985 [ 3318.272802] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3318.294668] LustreError: 115749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3318.712651] Lustre: server umount lustre-MDT0001 complete [ 3329.672852] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 3332.276128] Key type lgssc unregistered [ 3332.495317] LNet: 119331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3332.503625] LNetError: 119331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3332.522149] LNet: Removed LNI 192.168.206.105@tcp [ 3333.164808] Key type .llcrypt unregistered [ 3333.166760] Key type ._llcrypt unregistered [ 3354.279902] Key type ._llcrypt registered [ 3354.287289] Key type .llcrypt registered [ 3354.361238] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 3355.587770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3355.631996] alg: No test for adler32 (adler32-zlib) [ 3356.678857] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 3356.932390] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 3358.583189] Key type lgssc registered [ 3359.590688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3366.007632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3366.029122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3367.509274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3371.134502] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3377.632845] LustreError: 120747:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3377.645884] LustreError: 120747:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3377.763470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3378.047302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3381.400651] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3387.083833] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3392.173857] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3398.699958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3399.014607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3404.616816] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3412.038952] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3412.334948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3418.976217] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3422.519155] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3422.522972] Lustre: Skipped 2 previous similar messages [ 3426.478194] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3426.485655] Lustre: Skipped 2 previous similar messages [ 3429.428442] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3429.457166] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3435.110374] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.5@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3435.130399] Lustre: Skipped 2 previous similar messages [ 3437.738446] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.5@tcp 192.168.0.[2,10]@tcp [ 3441.087443] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.5/24@tcp in log lustre-MDT0000 [ 3441.096933] Lustre: Skipped 2 previous similar messages [ 3444.256986] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.5/24@tcp [ 3444.265063] Lustre: Skipped 1 previous similar message [ 3458.532145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3458.538119] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3458.543438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3458.550896] Lustre: Skipped 1 previous similar message [ 3461.759552] Lustre: server umount lustre-OST0000 complete [ 3465.185898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3465.195122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3465.214328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3465.219907] Lustre: Skipped 1 previous similar message [ 3468.769880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3468.771225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3471.284203] Lustre: server umount lustre-MDT0000 complete [ 3473.888328] LustreError: 120748:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3473.915455] LustreError: 120748:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3474.560382] LustreError: 120729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844808 with bad export cookie 18000205322411154816 [ 3474.565152] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3474.574398] LustreError: 120729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3474.872839] Lustre: server umount lustre-MDT0001 complete [ 3485.640314] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 3488.913103] Key type lgssc unregistered [ 3489.194516] LNet: 125540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3489.204712] LNetError: 125540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3489.232088] LNet: Removed LNI 192.168.206.105@tcp [ 3490.146608] Key type .llcrypt unregistered [ 3490.153212] Key type ._llcrypt unregistered [ 3515.755124] Key type ._llcrypt registered [ 3515.762142] Key type .llcrypt registered [ 3515.991089] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 3518.669725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3518.743547] alg: No test for adler32 (adler32-zlib) [ 3520.232275] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 3520.815540] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 3522.551846] Key type lgssc registered [ 3525.100832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3537.628790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3537.666710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3539.215708] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3539.244081] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.5/24@tcp [ 3539.273129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3543.359856] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3549.667435] LustreError: 127082:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3549.685429] LustreError: 127082:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3551.828692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3552.160987] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3552.192330] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.5/24@tcp [ 3552.250664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3557.028325] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3564.445438] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3569.755803] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3575.629193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3576.074484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3581.595254] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3584.260746] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3589.573057] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3599.847191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3599.852557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3599.858462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3599.883919] Lustre: Skipped 1 previous similar message [ 3604.963273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3604.972796] Lustre: Skipped 2 previous similar messages [ 3605.409597] Lustre: server umount lustre-OST0000 complete [ 3610.081342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3610.089829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3610.098046] Lustre: Skipped 1 previous similar message [ 3610.107630] Lustre: Skipped 1 previous similar message [ 3615.014363] Lustre: server umount lustre-MDT0000 complete [ 3615.204528] LustreError: 127082:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3615.238595] LustreError: 127082:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3618.364970] LustreError: 127065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844952 with bad export cookie 18039703204808534873 [ 3618.380078] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3618.667682] Lustre: server umount lustre-MDT0001 complete [ 3627.192744] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 3630.013849] Key type lgssc unregistered [ 3630.272539] LNet: 130676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3630.283840] LNetError: 130676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3630.304893] LNet: Removed LNI 192.168.206.105@tcp [ 3630.985749] Key type .llcrypt unregistered [ 3630.989700] Key type ._llcrypt unregistered [ 3654.728599] Key type ._llcrypt registered [ 3654.732419] Key type .llcrypt registered [ 3654.833662] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 3668.768404] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 3669.623879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3669.689378] alg: No test for adler32 (adler32-zlib) [ 3670.649234] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 3670.859750] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 3672.591571] Key type lgssc registered [ 3673.565748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3705.303864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3705.334894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3706.609896] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3706.660600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3706.759118] Lustre: lustre-MDT0000: new disk, initializing [ 3706.896722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3706.911382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3712.127504] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3723.754905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3723.955925] Lustre: 134073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3724.016857] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3724.020885] Lustre: Skipped 1 previous similar message [ 3724.130914] Lustre: lustre-MDT0001: new disk, initializing [ 3724.216733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3724.260404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3724.270259] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3729.112714] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3738.437225] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3744.638872] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3751.851455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3752.109347] Lustre: lustre-OST0000: new disk, initializing [ 3752.118232] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3752.130122] Lustre: 136043:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3752.205506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3759.554814] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3761.708723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3761.717996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3761.811574] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3769.502541] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3774.677496] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3774.925688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3779.809869] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3779.975700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3782.624538] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3782.626969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3782.637151] LustreError: Skipped 1 previous similar message [ 3782.642204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3782.653461] Lustre: Skipped 1 previous similar message [ 3782.676049] Lustre: Skipped 1 previous similar message [ 3787.746630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3788.247879] Lustre: server umount lustre-OST0000 complete [ 3792.875141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3792.881232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3792.890317] Lustre: Skipped 1 previous similar message [ 3792.895436] Lustre: Skipped 1 previous similar message [ 3797.948256] Lustre: server umount lustre-MDT0000 complete [ 3801.984892] LustreError: 134066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845135 with bad export cookie 16989380982372358273 [ 3801.986963] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3801.996429] LustreError: 134066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3802.385751] Lustre: server umount lustre-MDT0001 complete [ 3810.738869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3811.279095] LustreError: 138258:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3811.300565] LustreError: 138258:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 3811.350799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3815.765215] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3816.418500] LustreError: 138259:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3821.540634] LustreError: 138258:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3824.871952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3825.258488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3830.082506] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3838.243823] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3843.657787] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3849.729507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3850.074353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3856.481058] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3864.464171] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3868.506655] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3868.709485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3873.625332] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3873.928721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3883.507918] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3883.663570] Lustre: lustre-OST0001: new disk, initializing [ 3883.667189] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3883.681041] Lustre: 141637:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3883.810674] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3889.073305] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3890.735988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3890.742436] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3890.825306] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3896.897464] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3897.031946] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3900.819772] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3901.064571] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3906.008145] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff986107686800.ost_server_uuid 50 [ 3907.768911] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff986107686800.ost_server_uuid in IDLE state after 0 sec [ 3917.737719] Lustre: server umount lustre-OST0001 complete [ 3921.377757] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3921.393654] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3921.430836] LustreError: 140413:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3922.915686] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3922.920092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3922.921234] LustreError: Skipped 2 previous similar messages [ 3922.958432] Lustre: Skipped 2 previous similar messages [ 3927.011504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3927.014441] LustreError: 140080:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3927.026037] Lustre: Skipped 1 previous similar message [ 3927.043307] LustreError: 140080:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3928.384232] Lustre: server umount lustre-OST0000 complete [ 3932.129754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3932.148437] Lustre: Skipped 3 previous similar messages [ 3937.254222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3937.273118] Lustre: Skipped 3 previous similar messages [ 3938.237281] Lustre: server umount lustre-MDT0000 complete [ 3941.973335] LustreError: 138240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845275 with bad export cookie 16989380982372359155 [ 3941.987953] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3942.312451] Lustre: server umount lustre-MDT0001 complete [ 3959.914362] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 3963.219177] Key type lgssc unregistered [ 3963.593228] LNet: 144642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3963.601737] LNetError: 144642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3963.618508] LNet: Removed LNI 192.168.206.105@tcp [ 3964.651719] Key type .llcrypt unregistered [ 3964.654172] Key type ._llcrypt unregistered [ 4006.416616] Key type ._llcrypt registered [ 4006.418691] Key type .llcrypt registered [ 4006.546380] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 4019.294603] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 4020.360096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4020.404534] alg: No test for adler32 (adler32-zlib) [ 4021.525867] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 4021.754863] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 4023.447454] Key type lgssc registered [ 4024.680060] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4069.234282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4069.263042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4070.536955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4070.571753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4070.728477] Lustre: lustre-MDT0000: new disk, initializing [ 4070.909829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4070.943736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4077.499086] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4089.753806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4089.844879] Lustre: 148779:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4089.912576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4089.916625] Lustre: Skipped 1 previous similar message [ 4089.977349] Lustre: lustre-MDT0001: new disk, initializing [ 4090.064061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4090.099894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4090.135537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4094.539476] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4103.435174] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4108.561653] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4114.799085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4115.002913] Lustre: lustre-OST0000: new disk, initializing [ 4115.008664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4115.014448] Lustre: 150749:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4115.098238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4120.972280] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4124.745043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4124.759916] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4124.905930] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4129.658590] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4133.728238] Lustre: Failing over lustre-OST0000 [ 4133.965609] Lustre: server umount lustre-OST0000 complete [ 4151.603498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4151.775302] Lustre: 147005:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787845469/real 1787845469] req@000000004dfd0342 x1874691730997248/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787845485 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4151.777233] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4151.794051] Lustre: 147005:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4151.882822] Lustre: lustre-OST0000: Not available for connect from 192.168.206.5@tcp (not set up) [ 4151.993357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4152.007610] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4156.973042] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4157.477547] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4157.478587] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4157.605649] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4158.560103] Lustre: 147004:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787845474/real 1787845474] req@00000000d4f8e618 x1874691730998400/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787845490 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4158.592219] Lustre: 147004:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4161.591837] Lustre: Failing over lustre-MDT0000 [ 4161.886248] Lustre: server umount lustre-MDT0000 complete [ 4162.025773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4162.035471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4162.051261] Lustre: Skipped 1 previous similar message [ 4162.061496] LustreError: 148792:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4162.091471] LustreError: 148792:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4163.553248] Lustre: 147004:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787845479/real 1787845479] req@000000000e8ac35d x1874691730999424/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787845495 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4163.581491] Lustre: 147004:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4167.199337] Lustre: 147002:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787845484/real 1787845484] req@000000008ed0e877 x1874691731000576/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787845500 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4167.228532] LustreError: 148786:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4167.258196] LustreError: 148786:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4172.331206] LustreError: 150493:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4172.348957] LustreError: 150493:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4177.457895] LustreError: 148786:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4177.493355] LustreError: 148786:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4178.913833] Lustre: 147004:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787845496/real 1787845496] req@00000000c5dd0ad3 x1874691731008768/t0(0) o400->MGC192.168.206.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787845512 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4178.944305] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4179.095419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4182.570776] LustreError: 150757:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4182.589972] LustreError: 150757:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 4189.171864] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x9334c97795b9642a [ 4189.184368] Lustre: MGC192.168.206.105@tcp: Connection restored to 0@lo (at 0@lo) [ 4189.189501] Lustre: Skipped 1 previous similar message [ 4189.597265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4189.695265] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4192.807968] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4194.816838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4194.835669] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4194.871629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4195.378761] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4205.029069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4205.052797] Lustre: Skipped 2 previous similar messages [ 4205.062518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4205.067641] Lustre: Skipped 1 previous similar message [ 4209.792334] Lustre: server umount lustre-OST0000 complete [ 4215.273142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4215.274275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4215.277864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4215.297587] Lustre: Skipped 2 previous similar messages [ 4219.699773] Lustre: server umount lustre-MDT0000 complete [ 4220.384324] LustreError: 148789:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4220.403154] LustreError: 148789:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 4223.295681] LustreError: 148774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845557 with bad export cookie 10607324537814606890 [ 4223.300939] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4223.306824] LustreError: 148774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4223.675488] Lustre: server umount lustre-MDT0001 complete [ 4234.026472] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 4236.438493] Key type lgssc unregistered [ 4236.679550] LNet: 154409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4236.690716] LNetError: 154409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4236.713578] LNet: Removed LNI 192.168.206.105@tcp [ 4237.589918] Key type .llcrypt unregistered [ 4237.594230] Key type ._llcrypt unregistered [ 4271.242844] Key type ._llcrypt registered [ 4271.245578] Key type .llcrypt registered [ 4271.340476] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 4272.290281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4272.303831] alg: No test for adler32 (adler32-zlib) [ 4273.351936] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 4273.578249] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 4275.247454] Key type lgssc registered [ 4276.264718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4284.012988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4284.033272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4285.432360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4289.572972] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4295.654127] LustreError: 155930:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4295.685469] LustreError: 155930:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4298.460144] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4300.769157] LustreError: 155931:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4301.803374] LustreError: 155959:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4301.811422] LustreError: 155959:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4301.818134] LustreError: 155959:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 4302.059201] Lustre: server umount lustre-MDT0000 complete [ 4316.317673] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 4324.780725] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 4359.213645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4359.405907] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4359.428254] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4359.500607] Lustre: lustre-MDT0000: new disk, initializing [ 4359.557446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4359.575235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4363.852696] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4374.219648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4374.375637] Lustre: 159261:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4374.452318] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4374.455213] Lustre: Skipped 1 previous similar message [ 4374.505667] Lustre: lustre-MDT0001: new disk, initializing [ 4374.585071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4374.625366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4374.644788] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4378.548963] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4388.538681] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4395.790918] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4404.297470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4404.674359] Lustre: lustre-OST0000: new disk, initializing [ 4404.681990] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4404.711964] Lustre: 161229:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4404.848402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4411.464589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4411.476243] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4411.656936] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4411.706678] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4422.066349] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4427.717838] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4429.803242] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:49:21 (1787845761) [ 4696.547662] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4696.552661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4696.568873] Lustre: Skipped 1 previous similar message [ 4696.579871] Lustre: Skipped 2 previous similar messages [ 4699.075282] Lustre: server umount lustre-MDT0000 complete [ 4701.665698] LustreError: 159268:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4701.691570] LustreError: 159268:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4703.217189] LustreError: 160057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846037 with bad export cookie 7517370270215240686 [ 4703.219530] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4703.226633] LustreError: 160057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4706.799112] LustreError: 161237:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4706.801103] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4706.834668] Lustre: Skipped 1 previous similar message [ 4706.842683] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4708.837712] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4709.784268] Lustre: server umount lustre-MDT0001 complete [ 4720.871458] Lustre: server umount lustre-OST0000 complete [ 4732.390586] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 4743.130872] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 4783.931324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4784.246440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4784.290771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4784.400505] Lustre: lustre-MDT0000: new disk, initializing [ 4784.502829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4784.543961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4790.324676] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4801.786756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4802.041791] Lustre: 165895:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4802.149174] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4802.160329] Lustre: Skipped 1 previous similar message [ 4802.288712] Lustre: lustre-MDT0001: new disk, initializing [ 4802.439163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4802.476055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4802.482747] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4807.643247] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4817.565624] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4823.429780] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4831.706895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4832.119544] Lustre: lustre-OST0000: new disk, initializing [ 4832.126962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4832.139795] Lustre: 167864:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4832.242504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4840.799390] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4841.035753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4841.048370] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4841.288638] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4851.972994] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4857.857409] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4858.192754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4864.766602] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4865.096085] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4872.164606] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4872.169737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4872.176805] Lustre: Skipped 1 previous similar message [ 4872.176876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4872.186882] Lustre: Skipped 1 previous similar message [ 4877.290663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4877.305665] Lustre: Skipped 1 previous similar message [ 4881.375273] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4881.618320] Lustre: server umount lustre-OST0000 complete [ 4887.522931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4887.545640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4900.319120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4900.633545] Lustre: server umount lustre-MDT0000 complete [ 4902.881679] LustreError: 165907:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4902.908938] LustreError: 165907:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4904.291582] LustreError: 165889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846238 with bad export cookie 7517370270215399516 [ 4904.298414] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4904.300807] LustreError: 165889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4904.540715] Lustre: server umount lustre-MDT0001 complete [ 4913.959874] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:57:25 (1787846245) [ 4933.215667] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 4944.454339] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 4988.905647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4989.208483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4989.242355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4989.363834] Lustre: lustre-MDT0000: new disk, initializing [ 4989.493494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4989.527611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4994.328504] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5005.194228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5005.314462] Lustre: 172625:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5005.571219] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5009.819679] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5018.954537] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5024.384852] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5031.262848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5031.498691] Lustre: lustre-OST0000: new disk, initializing [ 5031.501460] Lustre: Skipped 1 previous similar message [ 5031.506672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5031.510746] Lustre: Skipped 2 previous similar messages [ 5031.515730] Lustre: 174605:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5037.118282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5037.137832] Lustre: Skipped 1 previous similar message [ 5037.148529] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5037.253206] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5037.749167] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5046.357141] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5058.018600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5058.030565] LustreError: Skipped 1 previous similar message [ 5058.034504] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5058.044038] Lustre: Skipped 1 previous similar message [ 5058.048328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5058.054455] Lustre: Skipped 6 previous similar messages [ 5068.263540] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5068.488391] Lustre: server umount lustre-OST0000 complete [ 5072.353023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5072.365022] LustreError: Skipped 1 previous similar message [ 5072.369511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5072.380563] Lustre: Skipped 1 previous similar message [ 5078.594319] Lustre: server umount lustre-MDT0000 complete [ 5082.518779] LustreError: 172616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846416 with bad export cookie 7517370270215400391 [ 5082.520244] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5082.532346] LustreError: 172616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5082.914128] Lustre: server umount lustre-MDT0001 complete [ 5091.935893] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:00:23 (1787846423) [ 5110.090345] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 5120.948849] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 5158.423481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5158.632958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5158.660511] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5158.735803] Lustre: lustre-MDT0000: new disk, initializing [ 5158.802924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5158.808788] Lustre: Skipped 2 previous similar messages [ 5158.822657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5163.886123] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5176.891587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5177.085113] Lustre: 178941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5177.417056] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5182.122278] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5191.683923] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5197.932925] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5205.188214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5205.507084] Lustre: 180919:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5212.263738] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5213.789050] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5213.901106] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5222.324517] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5229.537503] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5229.540929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5229.545285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5229.556061] Lustre: Skipped 2 previous similar messages [ 5229.571971] Lustre: Skipped 10 previous similar messages [ 5243.363975] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5243.629376] Lustre: server umount lustre-OST0000 complete [ 5248.999475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5253.966027] Lustre: server umount lustre-MDT0000 complete [ 5255.138953] LustreError: 178954:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5255.154447] LustreError: 178954:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5258.511509] LustreError: 178934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846592 with bad export cookie 7517370270215401483 [ 5258.515082] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5258.524434] LustreError: 178934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5259.072618] Lustre: server umount lustre-MDT0001 complete [ 5271.938276] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 5274.743382] Key type lgssc unregistered [ 5275.011475] LNet: 182780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5275.016838] LNetError: 182780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5275.041819] LNet: Removed LNI 192.168.206.105@tcp [ 5275.918588] Key type .llcrypt unregistered [ 5275.921464] Key type ._llcrypt unregistered [ 5300.637940] Key type ._llcrypt registered [ 5300.641806] Key type .llcrypt registered [ 5300.769156] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 5301.939833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5301.965728] alg: No test for adler32 (adler32-zlib) [ 5303.064397] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 5303.350314] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 5305.167155] Key type lgssc registered [ 5306.894568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5317.647257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5317.695063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5319.500574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5324.977464] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5329.888878] LustreError: 184199:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5329.914475] LustreError: 184199:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5335.012837] LustreError: 184198:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5335.899859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5336.494122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5341.605816] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5351.137235] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5356.625905] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5363.214032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5363.464653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5364.457970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5369.392464] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5376.715149] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5400.544823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5400.557615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5400.579191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5402.596179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5402.601300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5402.602365] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5404.024068] Lustre: server umount lustre-OST0000 complete [ 5408.227037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5408.237291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5408.249225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5408.254676] Lustre: Skipped 1 previous similar message [ 5412.836358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5412.839531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5414.419325] Lustre: server umount lustre-MDT0000 complete [ 5417.954533] LustreError: 185183:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5417.977747] LustreError: 185183:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5418.836552] LustreError: 184179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846752 with bad export cookie 5962844100310050454 [ 5418.840324] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5418.857860] LustreError: 184179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5419.356291] Lustre: server umount lustre-MDT0001 complete [ 5430.759962] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 5434.164826] Key type lgssc unregistered [ 5434.615713] LNet: 187759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5434.622963] LNetError: 187759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5434.667254] LNet: Removed LNI 192.168.206.105@tcp [ 5435.955842] Key type .llcrypt unregistered [ 5435.958071] Key type ._llcrypt unregistered [ 5465.419472] Key type ._llcrypt registered [ 5465.422334] Key type .llcrypt registered [ 5465.536574] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 5467.241755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5467.274088] alg: No test for adler32 (adler32-zlib) [ 5468.411418] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 5468.656694] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 5470.423241] Key type lgssc registered [ 5471.748425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5480.226475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5480.263526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5481.787718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5486.657115] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5492.194449] LustreError: 189182:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5492.236768] LustreError: 189182:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5497.316336] LustreError: 189181:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5498.124896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5498.744286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5504.240053] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5513.945331] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5521.144426] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5528.909393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5529.179248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5537.956701] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5546.166778] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5552.761339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5558.242280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5558.251632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5558.266092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5558.288156] Lustre: Skipped 1 previous similar message [ 5558.304808] Lustre: Skipped 1 previous similar message [ 5563.368215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5563.388186] Lustre: Skipped 2 previous similar messages [ 5568.489541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5568.503670] Lustre: Skipped 2 previous similar messages [ 5572.063355] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5572.321768] Lustre: server umount lustre-OST0000 complete [ 5580.474545] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff98610378f800.ost_server_uuid 50 [ 5582.979361] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5597.838427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5597.862699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5597.878597] Lustre: Skipped 2 previous similar messages [ 5604.262521] Lustre: server umount lustre-MDT0000 complete [ 5604.323249] LustreError: 189181:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5604.349621] LustreError: 189181:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5608.728220] LustreError: 189162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846942 with bad export cookie 7409392417028161718 [ 5608.752237] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5608.754683] LustreError: 189162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5609.473383] Lustre: server umount lustre-MDT0001 complete [ 5619.162033] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 12:09:11 (1787846951) [ 5626.277295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5626.858926] LustreError: 193102:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5627.000807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5631.554503] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5632.480803] LustreError: 193103:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5640.526729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5640.856106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5645.205248] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5653.547220] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5659.289624] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5666.557207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5666.954268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5672.134118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5673.658837] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5684.679787] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5693.446380] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5693.665623] Lustre: lustre-OST0001: new disk, initializing [ 5693.670384] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5693.678172] Lustre: 195940:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5693.788216] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5700.893736] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5701.275023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5701.283759] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5701.503186] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5712.760985] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5716.450716] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5716.469443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5716.485802] Lustre: Skipped 1 previous similar message [ 5716.491875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5716.497347] Lustre: Skipped 2 previous similar messages [ 5727.206871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5727.219821] Lustre: Skipped 6 previous similar messages [ 5730.271166] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5730.453177] Lustre: server umount lustre-OST0000 complete [ 5732.329989] LustreError: 194925:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5732.382758] LustreError: 194925:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5738.332962] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5738.648058] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5744.436499] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5744.764225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5749.807374] LustreError: 194922:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5749.842570] LustreError: 194922:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 5763.040805] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5763.045440] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5763.048675] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5763.055445] Lustre: Skipped 2 previous similar messages [ 5763.083031] Lustre: Skipped 2 previous similar messages [ 5764.706224] Lustre: server umount lustre-OST0001 complete [ 5773.280163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5773.301172] Lustre: Skipped 1 previous similar message [ 5775.137356] Lustre: server umount lustre-MDT0000 complete [ 5778.918788] LustreError: 193082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847112 with bad export cookie 7409392417028162789 [ 5778.920681] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5778.927936] LustreError: 193082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5779.273106] Lustre: server umount lustre-MDT0001 complete [ 5813.642922] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 12:12:25 (1787847145) [ 5821.737262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5822.287763] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5822.361752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5822.602070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5827.407411] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5839.215598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5839.347596] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5839.388389] Lustre: 199386:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5844.232677] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5851.925247] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5857.904912] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5864.663633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5864.861835] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5871.028047] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5875.287125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5878.043584] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5885.361043] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5885.461103] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5892.380391] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5900.757858] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5905.391195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5905.399904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5905.404043] Lustre: Skipped 2 previous similar messages [ 5909.790277] Lustre: server umount lustre-OST0000 complete [ 5910.525842] LustreError: 201232:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5910.553181] LustreError: 201232:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [ 5925.858427] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5925.863249] LustreError: Skipped 1 previous similar message [ 5925.868920] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5925.891832] Lustre: Skipped 1 previous similar message [ 5930.464661] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5931.787834] Lustre: server umount lustre-OST0001 complete [ 5942.615532] Lustre: server umount lustre-MDT0000 complete [ 5947.181128] LustreError: 199379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847280 with bad export cookie 7409392417028164196 [ 5947.191462] LustreError: 199379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5947.192189] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5947.509891] Lustre: server umount lustre-MDT0001 complete [ 5972.978491] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 12:15:05 (1787847305) [ 5990.652296] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 5999.823940] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 6035.329182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6035.560856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6035.586929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6035.659555] Lustre: lustre-MDT0000: new disk, initializing [ 6035.752463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6035.767529] Lustre: Skipped 3 previous similar messages [ 6035.800418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6040.548258] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6050.312873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6050.407867] Lustre: 207487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6050.482869] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6050.488399] Lustre: Skipped 1 previous similar message [ 6050.551331] Lustre: lustre-MDT0001: new disk, initializing [ 6050.666696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6050.676031] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6054.261335] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6063.341638] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6069.556222] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6075.922568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6076.253214] Lustre: lustre-OST0000: new disk, initializing [ 6076.257804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6076.265798] Lustre: 209458:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6081.981262] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6086.176249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6086.185539] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6086.310888] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6091.050206] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6094.863260] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6095.027094] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6099.438539] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6099.707461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6101.987209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6101.995741] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6102.004832] Lustre: Skipped 3 previous similar messages [ 6102.011492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6102.016780] Lustre: Skipped 8 previous similar messages [ 6107.967373] Lustre: server umount lustre-OST0000 complete [ 6127.071968] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6127.286193] Lustre: server umount lustre-MDT0000 complete [ 6127.593062] LustreError: 208494:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6127.611965] LustreError: 208494:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 14 previous similar messages [ 6131.371325] LustreError: 207479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847465 with bad export cookie 7409392417028165757 [ 6131.378811] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6131.398612] LustreError: 207479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6131.908782] Lustre: server umount lustre-MDT0001 complete [ 6140.967235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6146.291713] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6155.515789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6160.437918] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6168.514467] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6174.655701] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6182.162719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6182.688539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6182.699244] Lustre: Skipped 4 previous similar messages [ 6188.994542] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6197.799234] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6202.966967] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6203.221505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6208.028479] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6208.241644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6211.557145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6211.558767] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6211.582942] Lustre: Skipped 4 previous similar messages [ 6211.601530] LustreError: Skipped 2 previous similar messages [ 6224.351127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6224.544303] Lustre: server umount lustre-OST0000 complete [ 6231.774870] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6236.748329] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6266.318358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6274.347793] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6276.970455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.993806] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.994277] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6277.034258] Lustre: Skipped 1 previous similar message [ 6282.568930] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6284.318858] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6292.451632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6292.465880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6292.467119] LustreError: Skipped 1 previous similar message [ 6292.480887] Lustre: Skipped 14 previous similar messages [ 6295.018066] Lustre: server umount lustre-OST0000 complete [ 6306.003654] Lustre: server umount lustre-MDT0000 complete [ 6307.810860] LustreError: 212678:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6307.828581] LustreError: 212678:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 6309.825471] LustreError: 213519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847643 with bad export cookie 7409392417028166632 [ 6309.837318] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6309.838867] LustreError: 213519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6310.156585] Lustre: server umount lustre-MDT0001 complete [ 6319.031212] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 12:20:51 (1787847651) [ 6326.625451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6333.597687] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6341.392214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6346.485879] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6354.475527] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6361.839341] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6369.477239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6375.739523] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6383.213380] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6387.936425] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6388.197176] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6392.622925] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6392.781927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6400.445081] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6400.649596] Lustre: lustre-OST0001: new disk, initializing [ 6400.658789] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6400.664129] Lustre: 221015:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6407.276585] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6410.340530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6410.346971] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6410.436477] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6416.853629] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6420.917403] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6421.093371] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6425.855096] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6426.178739] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6431.202110] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6431.208169] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6431.216607] LustreError: Skipped 1 previous similar message [ 6431.245163] Lustre: Skipped 5 previous similar messages [ 6434.594103] Lustre: server umount lustre-OST0001 complete [ 6441.833348] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6442.092744] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6447.015711] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6447.269882] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6476.027504] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6476.365388] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6476.370921] Lustre: Skipped 5 previous similar messages [ 6481.417626] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6481.429907] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6481.437740] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6481.466415] Lustre: Skipped 1 previous similar message [ 6482.451353] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6490.096787] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6491.743908] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6499.797558] Lustre: server umount lustre-OST0001 complete [ 6522.695592] Lustre: server umount lustre-OST0000 complete [ 6541.279140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6543.424552] Lustre: server umount lustre-MDT0000 complete [ 6547.480869] LustreError: 217626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847881 with bad export cookie 7409392417028167605 [ 6547.492335] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6554.026375] Lustre: server umount lustre-MDT0001 complete [ 6577.393726] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 12:25:09 (1787847909) [ 6584.401169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6584.657523] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6584.707842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6589.234693] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6597.433632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6597.524859] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6597.553899] Lustre: 226272:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6602.393815] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6610.453414] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6617.046978] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6623.801856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6624.014769] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6630.881182] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6639.601575] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6652.142355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6652.358607] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6659.443103] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6666.911905] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6668.617213] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6674.004912] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6674.281966] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6679.865765] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6680.105116] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6686.398453] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98610308a800.ost_server_uuid 50 [ 6688.022598] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff98610308a800.ost_server_uuid in IDLE state after 0 sec [ 6689.197328] Lustre: Permanently deactivating lustre-OST0001 [ 6689.208368] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6693.158554] Lustre: Permanently reactivating lustre-OST0001 [ 6700.667718] Lustre: server umount lustre-OST0001 complete [ 6701.541494] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6701.551875] LustreError: Skipped 2 previous similar messages [ 6701.554876] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6701.572321] Lustre: Skipped 6 previous similar messages [ 6701.582931] LustreError: 228607:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6701.597529] LustreError: 228607:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 37 previous similar messages [ 6705.633378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6705.640283] Lustre: Skipped 18 previous similar messages [ 6710.864734] Lustre: server umount lustre-OST0000 complete [ 6722.165899] Lustre: server umount lustre-MDT0000 complete [ 6726.909611] LustreError: 229144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848060 with bad export cookie 7409392417028168893 [ 6726.916321] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6726.920892] LustreError: 229144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6727.386447] Lustre: server umount lustre-MDT0001 complete [ 6754.548583] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 12:28:06 (1787848086) [ 6763.317438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6763.764438] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6763.805602] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6763.830136] Lustre: Skipped 5 previous similar messages [ 6769.181407] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6778.644261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6778.797142] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6778.845668] Lustre: 232961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6783.944086] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6792.119814] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6798.508646] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6806.574810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6806.771386] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6806.798989] Lustre: Permanently deactivating lustre-OST0000 [ 6806.816584] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6813.411863] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6815.779492] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6822.326712] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6831.097634] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6831.238290] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6837.056711] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6839.238991] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6848.138779] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6850.855866] Lustre: Permanently reactivating lustre-OST0000 [ 6850.875536] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6850.879820] Lustre: Skipped 2 previous similar messages [ 6857.819278] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6857.819498] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 42 seconds. I think it's dead, and I am evicting it. exp 00000000879a63f9, cur 1787848191 deadline 1787848179 last 1787848149 [ 6857.838923] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6857.881025] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6857.983176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6867.836662] Lustre: server umount lustre-OST0001 complete [ 6886.367935] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6886.696779] Lustre: server umount lustre-OST0000 complete [ 6899.133170] Lustre: server umount lustre-MDT0000 complete [ 6902.895020] LustreError: 232954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848236 with bad export cookie 7409392417028170538 [ 6902.904201] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6902.916639] LustreError: 232954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6903.318410] Lustre: server umount lustre-MDT0001 complete [ 6915.123630] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 6917.491475] Key type lgssc unregistered [ 6917.796765] LNet: 237847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6917.807654] LNetError: 237847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6917.831248] LNet: Removed LNI 192.168.206.105@tcp [ 6918.560246] Key type .llcrypt unregistered [ 6918.561915] Key type ._llcrypt unregistered [ 6941.410137] Key type ._llcrypt registered [ 6941.414698] Key type .llcrypt registered [ 6941.590617] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 6943.049957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6943.087147] alg: No test for adler32 (adler32-zlib) [ 6944.262770] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 6944.501222] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 6946.215177] Key type lgssc registered [ 6947.622206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6958.393778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6958.447411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6959.936043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6964.728921] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6969.317939] LustreError: 239316:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6969.331499] LustreError: 239316:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6972.784112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6972.924086] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6972.964352] Lustre: 239303:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6972.981477] Lustre: Permanently deactivating lustre-MDT0001 [ 6973.009778] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6973.327575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6973.399127] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6973.418884] LustreError: 239347:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6973.463198] LustreError: 239347:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6973.474829] LustreError: 239347:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 1, failed: rc = -108 [ 6977.669729] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6986.204469] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6992.320848] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6999.436063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6999.846774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7004.976986] LustreError: 241159:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7005.000579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7006.324400] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7010.277063] LustreError: 241159:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7013.906715] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7015.396985] LustreError: 241156:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7018.490959] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7021.264490] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7021.532980] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7026.683814] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7026.700909] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7028.425811] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7036.772632] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7040.627673] Lustre: Permanently reactivating lustre-MDT0001 [ 7040.662990] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7040.676746] Lustre: Skipped 2 previous similar messages [ 7043.416593] Lustre: lustre-MDT0001: Client 79767b63-973b-4a8e-98a5-f2339567176d (at 192.168.206.5@tcp) reconnecting [ 7050.196423] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7052.105993] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7058.313426] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7058.571413] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7063.462868] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7063.685488] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7066.534749] Lustre: Permanently deactivating lustre-MDT0001 [ 7066.559749] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7066.564501] Lustre: Skipped 2 previous similar messages [ 7074.411670] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7082.982511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7082.988665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7082.990439] Lustre: Skipped 1 previous similar message [ 7083.003413] Lustre: Skipped 3 previous similar messages [ 7083.741509] Lustre: server umount lustre-MDT0000 complete [ 7087.541230] LustreError: 239296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848421 with bad export cookie 13776930387560835906 [ 7087.545088] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7087.550279] LustreError: 239296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7088.102586] Lustre: server umount lustre-MDT0001 complete [ 7094.514757] Lustre: server umount lustre-OST0000 complete [ 7106.554502] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:33:58 (1787848438) [ 7118.808852] Lustre: server umount lustre-OST0001 complete [ 7126.393416] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 7136.344591] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 7173.187606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7173.464487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7173.473644] Lustre: Skipped 2 previous similar messages [ 7173.495230] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7173.553708] Lustre: lustre-MDT0000: new disk, initializing [ 7173.614802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7173.628493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7177.736672] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7189.143922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7189.360505] Lustre: 247575:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7189.425438] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7189.428584] Lustre: Skipped 1 previous similar message [ 7189.498688] Lustre: lustre-MDT0001: new disk, initializing [ 7189.627032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7189.657555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7189.669531] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7193.769382] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7202.994817] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7208.853882] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7215.747744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7216.083645] Lustre: lustre-OST0000: new disk, initializing [ 7216.087755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7216.095887] Lustre: 249546:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7216.180252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7222.872515] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7223.844102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7223.854059] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7223.997176] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7231.777654] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7234.004365] LustreError: 247583:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7242.867440] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7243.083412] Lustre: lustre-OST0001: new disk, initializing [ 7243.089858] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7243.099716] Lustre: 250659:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7244.007118] LustreError: 247583:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7244.033830] LustreError: 247581:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7248.918898] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7253.091473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7253.128342] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7253.243918] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7254.040687] LustreError: 247581:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7258.876366] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7260.387670] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7263.722634] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7263.724429] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7263.724820] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7263.743228] Lustre: Skipped 3 previous similar messages [ 7263.760771] Lustre: Skipped 1 previous similar message [ 7266.381378] Lustre: lustre-OST0001: Not available for connect from 192.168.206.5@tcp (stopping) [ 7268.578560] Lustre: server umount lustre-OST0001 complete [ 7268.847215] LustreError: 249530:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7268.865274] LustreError: 249530:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 7279.078360] LustreError: 251116:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7279.091545] LustreError: 251116:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 7289.328528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7289.340788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7289.341205] Lustre: Skipped 1 previous similar message [ 7289.368173] Lustre: Skipped 1 previous similar message [ 7292.513308] Lustre: server umount lustre-OST0000 complete [ 7296.992354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7297.014354] LustreError: Skipped 1 previous similar message [ 7297.021744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7297.041541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7302.564179] Lustre: server umount lustre-MDT0000 complete [ 7304.675793] LustreError: 249206:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7304.702349] LustreError: 249206:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 7306.247059] LustreError: 248371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848640 with bad export cookie 13776930387560838720 [ 7306.257234] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7306.259861] LustreError: 248371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7312.658615] Lustre: server umount lustre-MDT0001 complete [ 7324.030977] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 7326.380857] Key type lgssc unregistered [ 7326.644850] LNet: 252698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7326.656293] LNetError: 252698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7326.681037] LNet: Removed LNI 192.168.206.105@tcp [ 7327.519056] Key type .llcrypt unregistered [ 7327.525084] Key type ._llcrypt unregistered [ 7366.408812] Key type ._llcrypt registered [ 7366.411854] Key type .llcrypt registered [ 7366.531481] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 7367.892453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7367.916046] alg: No test for adler32 (adler32-zlib) [ 7369.096482] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 7369.400157] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 7371.151185] Key type lgssc registered [ 7372.619457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7380.934737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7380.949631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7382.221692] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7382.255180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7382.428093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7386.647634] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7395.610065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7395.758504] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7395.816801] Lustre: 254908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7396.166681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7400.951360] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7409.104644] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7415.225418] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7423.311528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7423.541526] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7423.782648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7429.315109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7430.797474] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7440.153616] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7500.768743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7500.779057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7500.799970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7501.281988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7501.284115] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7501.321316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7501.328839] Lustre: Skipped 1 previous similar message [ 7503.229299] Lustre: server umount lustre-OST0000 complete [ 7507.952618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7523.659398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7524.262115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7526.305789] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7526.330712] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7532.331793] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7541.701132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7541.740669] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7542.120359] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7587.810545] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7587.830525] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7587.858532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7589.261124] Lustre: server umount lustre-OST0000 complete [ 7592.931371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7592.942717] Lustre: Skipped 1 previous similar message [ 7592.947598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7592.956361] Lustre: Skipped 2 previous similar messages [ 7598.049852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7598.065568] Lustre: Skipped 1 previous similar message [ 7599.046121] Lustre: server umount lustre-MDT0000 complete [ 7603.237116] LustreError: 254901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848937 with bad export cookie 8990699104555328519 [ 7603.244497] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7603.248167] LustreError: 254901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7603.607729] Lustre: server umount lustre-MDT0001 complete [ 7617.109515] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 7619.802910] Key type lgssc unregistered [ 7620.070622] LNet: 260334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7620.080761] LNetError: 260334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7620.102223] LNet: Removed LNI 192.168.206.105@tcp [ 7620.985166] Key type .llcrypt unregistered [ 7620.987282] Key type ._llcrypt unregistered [ 7644.552635] Key type ._llcrypt registered [ 7644.554505] Key type .llcrypt registered [ 7644.683401] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 7645.916966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7645.982314] alg: No test for adler32 (adler32-zlib) [ 7647.032636] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 7647.431454] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 7649.127261] Key type lgssc registered [ 7650.382410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7657.848592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7657.872542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7659.534908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7663.848705] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7669.729473] LustreError: 261752:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7669.742576] LustreError: 261752:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7671.960791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7672.305656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7676.470430] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7684.301735] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7690.121446] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7697.343380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7697.804280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7704.939855] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7704.940729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7712.799149] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7735.777210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7735.789759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7735.795243] Lustre: Skipped 1 previous similar message [ 7740.896755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7740.909301] Lustre: Skipped 2 previous similar messages [ 7741.909629] Lustre: server umount lustre-OST0000 complete [ 7746.025729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7746.025729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7746.027157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7746.054897] Lustre: Skipped 1 previous similar message [ 7751.137195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7756.262087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7756.272944] Lustre: Skipped 2 previous similar messages [ 7760.351133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7760.753485] Lustre: server umount lustre-MDT0000 complete [ 7761.376492] LustreError: 261752:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7761.393383] LustreError: 261752:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7764.843295] LustreError: 265307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849098 with bad export cookie 3543332898797225237 [ 7764.854388] LustreError: 265307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7764.856111] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7765.513144] Lustre: server umount lustre-MDT0001 complete [ 7777.468653] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 7780.315253] Key type lgssc unregistered [ 7780.714866] LNet: 266062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7780.724995] LNetError: 266062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7781.803846] LNet: Removed LNI 192.168.206.105@tcp [ 7782.906313] Key type .llcrypt unregistered [ 7782.910723] Key type ._llcrypt unregistered [ 7802.713291] Key type ._llcrypt registered [ 7802.719072] Key type .llcrypt registered [ 7802.825781] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 7803.911171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7804.003660] alg: No test for adler32 (adler32-zlib) [ 7805.023877] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 7805.235821] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 7806.911268] Key type lgssc registered [ 7808.168570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7815.177407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7815.193153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7816.763743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7820.908038] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7826.921360] LustreError: 267300:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7826.955207] LustreError: 267300:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7828.414045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7828.915708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7833.350891] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7841.045440] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7846.930623] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7854.075460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7854.409622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7860.543302] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7861.623392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7868.514317] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7882.217461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7882.224040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7882.244179] Lustre: Skipped 1 previous similar message [ 7882.261552] Lustre: Skipped 1 previous similar message [ 7887.330763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7887.339838] Lustre: Skipped 1 previous similar message [ 7887.815236] Lustre: server umount lustre-OST0000 complete [ 7892.451369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7892.453658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7892.468194] Lustre: Skipped 1 previous similar message [ 7892.488363] Lustre: Skipped 1 previous similar message [ 7897.571852] LustreError: 267301:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7897.669491] Lustre: server umount lustre-MDT0000 complete [ 7901.186455] LustreError: 267280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849235 with bad export cookie 3435461763229914927 [ 7901.194363] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7901.199163] LustreError: 267280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7901.515663] Lustre: server umount lustre-MDT0001 complete [ 7910.720108] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 7913.634336] Key type lgssc unregistered [ 7913.894495] LNet: 270985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7913.901608] LNetError: 270985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7913.918992] LNet: Removed LNI 192.168.206.105@tcp [ 7914.821186] Key type .llcrypt unregistered [ 7914.823480] Key type ._llcrypt unregistered [ 7945.859499] Key type ._llcrypt registered [ 7945.862463] Key type .llcrypt registered [ 7946.041685] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 7947.145420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7947.251727] alg: No test for adler32 (adler32-zlib) [ 7948.504752] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 7948.791293] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 7950.551160] Key type lgssc registered [ 7951.832751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7958.936714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7958.960200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7960.405364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7965.423676] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7970.788443] LustreError: 272403:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7970.812957] LustreError: 272403:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7972.715840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7973.022329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7976.827368] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7983.547716] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7988.976809] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7995.536024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7996.000935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8001.533176] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 8005.242790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8007.845243] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8031.218289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8031.224890] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8031.232532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8031.243621] Lustre: Skipped 1 previous similar message [ 8036.327046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8036.343250] Lustre: Skipped 2 previous similar messages [ 8036.600356] Lustre: server umount lustre-OST0000 complete [ 8041.442028] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8041.459368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8041.471371] Lustre: Skipped 1 previous similar message [ 8046.563562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8046.571516] Lustre: Skipped 1 previous similar message [ 8046.971858] Lustre: server umount lustre-MDT0000 complete [ 8050.591870] LustreError: 272382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849384 with bad export cookie 11791394923418568710 [ 8050.595361] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8050.610743] LustreError: 272382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8051.167605] Lustre: server umount lustre-MDT0001 complete [ 8063.120449] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 8065.915363] Key type lgssc unregistered [ 8066.305716] LNet: 276709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8066.316671] LNetError: 276709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8066.349251] LNet: Removed LNI 192.168.206.105@tcp [ 8067.334781] Key type .llcrypt unregistered [ 8067.337091] Key type ._llcrypt unregistered [ 8085.741265] Key type ._llcrypt registered [ 8085.743808] Key type .llcrypt registered [ 8085.847636] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 8086.837446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8086.945574] alg: No test for adler32 (adler32-zlib) [ 8087.969188] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 8088.256981] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 8089.959283] Key type lgssc registered [ 8091.107412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8099.221845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8099.254146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8100.657221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8105.714490] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 8111.075723] LustreError: 277947:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8111.097989] LustreError: 277947:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8114.646736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8115.031457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8119.979865] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 8128.542374] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8135.500865] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8142.482035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8142.856173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8145.924913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8149.667254] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 8158.453383] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8174.562640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8174.606390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8174.616504] Lustre: Skipped 1 previous similar message [ 8179.049750] Lustre: server umount lustre-OST0000 complete [ 8186.856223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8186.871625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8186.924895] Lustre: Skipped 1 previous similar message [ 8186.934803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8189.922511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8189.925586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8189.951505] Lustre: Skipped 1 previous similar message [ 8192.467027] Lustre: server umount lustre-MDT0000 complete [ 8195.050031] LustreError: 277956:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8195.073352] LustreError: 277956:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8196.064983] LustreError: 277939:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8196.081443] LustreError: 277939:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8197.410151] LustreError: 277913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849531 with bad export cookie 18056930076535999004 [ 8197.412212] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8197.422360] LustreError: 277913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8197.898234] Lustre: server umount lustre-MDT0001 complete [ 8211.040841] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 8214.187631] Key type lgssc unregistered [ 8214.599614] LNet: 281654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8214.609630] LNetError: 281654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8214.630794] LNet: Removed LNI 192.168.206.105@tcp [ 8215.697196] Key type .llcrypt unregistered [ 8215.699503] Key type ._llcrypt unregistered [ 8229.777230] Key type ._llcrypt registered [ 8229.778652] Key type .llcrypt registered [ 8229.869770] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8248.011599] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 8263.286654] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 8264.160278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8264.188083] alg: No test for adler32 (adler32-zlib) [ 8265.345858] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 8265.633825] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 8267.351156] Key type lgssc registered [ 8268.340547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8302.317657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8302.354751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8303.596869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8303.655057] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8303.786091] Lustre: lustre-MDT0000: new disk, initializing [ 8303.907212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8303.939054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8308.207264] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 8318.379913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8318.487080] Lustre: 285209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8318.532457] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8318.536943] Lustre: Skipped 1 previous similar message [ 8318.600399] Lustre: lustre-MDT0001: new disk, initializing [ 8318.689299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8318.732160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8318.745902] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8322.789885] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 8332.201509] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8338.028833] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8344.886921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8345.301708] Lustre: lustre-OST0000: new disk, initializing [ 8345.308914] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8345.325250] Lustre: 287176:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8345.438926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8351.248722] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 8355.410371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8355.420453] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8355.550553] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8361.093503] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8366.614351] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8366.880648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8371.892965] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8372.173504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8376.289111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8376.294069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8376.300448] LustreError: Skipped 1 previous similar message [ 8376.305100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8376.311539] Lustre: Skipped 1 previous similar message [ 8376.320986] Lustre: Skipped 1 previous similar message [ 8380.337558] Lustre: server umount lustre-OST0000 complete [ 8385.504251] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8385.512065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8385.530890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8386.531342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8386.545836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8386.550365] Lustre: Skipped 1 previous similar message [ 8390.035053] Lustre: server umount lustre-MDT0000 complete [ 8393.793593] LustreError: 285202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849727 with bad export cookie 2179649397627246763 [ 8393.799060] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8393.805184] LustreError: 285202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8394.211403] Lustre: server umount lustre-MDT0001 complete [ 8402.875446] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 12:55:34 (1787849734) [ 8409.740934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8410.319276] LustreError: 289589:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8410.333618] LustreError: 289589:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 8410.382173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8414.988353] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 8415.721758] LustreError: 289590:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8420.833112] LustreError: 289589:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8424.240840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8424.814916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8429.573310] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 8438.185773] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8444.658187] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8452.045318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8452.467810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8458.606481] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 8465.742022] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8480.224121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8480.227887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8480.232612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8480.234155] Lustre: Skipped 1 previous similar message [ 8480.240897] Lustre: Skipped 1 previous similar message [ 8484.756787] Lustre: server umount lustre-OST0000 complete [ 8490.466832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8490.474390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8490.484211] Lustre: Skipped 1 previous similar message [ 8490.496833] Lustre: Skipped 1 previous similar message [ 8494.880106] Lustre: server umount lustre-MDT0000 complete [ 8495.586511] LustreError: 291127:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8495.603117] LustreError: 291127:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8498.670234] LustreError: 292396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849832 with bad export cookie 2179649397627247645 [ 8498.670821] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8498.675969] LustreError: 292396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8498.967136] Lustre: server umount lustre-MDT0001 complete [ 8510.287376] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 8512.897235] Key type lgssc unregistered [ 8513.208468] LNet: 293152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8513.214384] LNetError: 293152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8513.237526] LNet: Removed LNI 192.168.206.105@tcp [ 8514.009299] Key type .llcrypt unregistered [ 8514.014495] Key type ._llcrypt unregistered [ 8537.057574] Key type ._llcrypt registered [ 8537.065458] Key type .llcrypt registered [ 8537.151439] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 8550.408421] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 8551.616852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8551.641301] alg: No test for adler32 (adler32-zlib) [ 8552.645751] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 8552.887919] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 8554.535281] Key type lgssc registered [ 8555.526522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8599.192433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8599.245658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8600.649154] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8600.714265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8600.824405] Lustre: lustre-MDT0000: new disk, initializing [ 8600.913857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8600.939724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8606.741871] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 8617.391387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8617.557621] Lustre: 296971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8617.617894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8617.623579] Lustre: Skipped 1 previous similar message [ 8617.739231] Lustre: lustre-MDT0001: new disk, initializing [ 8617.858063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8617.952665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8617.970234] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8622.202588] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 8632.207528] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8638.504269] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8644.788850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8645.027926] Lustre: lustre-OST2710: new disk, initializing [ 8645.036684] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8645.042765] Lustre: 298940:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8645.105234] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8651.489728] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 8653.360093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8653.388848] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8653.542483] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8663.002238] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8671.111447] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8671.289119] Lustre: lustre-OST03e8: new disk, initializing [ 8671.297843] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8671.311771] Lustre: 300005:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8671.482246] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8678.726759] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 8680.572419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8680.586340] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8680.731323] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8686.323042] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8694.151813] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8694.347872] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8698.300984] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8698.507924] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8703.218917] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8703.459994] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8707.443809] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8707.673065] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8715.234642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8715.242479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8715.282132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8716.771397] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8716.783553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8716.796382] Lustre: Skipped 2 previous similar messages [ 8716.818986] Lustre: Skipped 3 previous similar messages [ 8718.607683] Lustre: server umount lustre-MDT0000 complete [ 8722.122135] LustreError: 297767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850055 with bad export cookie 7230724829328560501 [ 8722.127816] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8722.135702] LustreError: 297767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8722.707615] Lustre: server umount lustre-MDT0001 complete [ 8736.317517] Lustre: server umount lustre-OST2710 complete [ 8750.301928] Lustre: server umount lustre-OST03e8 complete [ 8770.249306] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 8779.116389] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 8816.624595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8817.075402] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8817.111414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8817.181259] Lustre: lustre-MDT0000: new disk, initializing [ 8817.335181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8817.362807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8821.622914] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 8833.264790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8833.449126] Lustre: 305264:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8833.527439] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8833.532204] Lustre: Skipped 1 previous similar message [ 8833.664586] Lustre: lustre-MDT0001: new disk, initializing [ 8833.809376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8833.863467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8833.881559] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8838.240953] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 8849.542148] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8856.186337] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8863.096095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8863.309667] Lustre: lustre-OST0000: new disk, initializing [ 8863.313210] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8863.321531] Lustre: 307237:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8863.410451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8868.899593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8868.908313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8868.989490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8869.707789] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 8879.571088] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8884.779757] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8885.078577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8890.878080] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8891.232668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8894.433469] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8894.437241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8894.444754] LustreError: Skipped 1 previous similar message [ 8894.457861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8894.472140] Lustre: Skipped 1 previous similar message [ 8894.500362] Lustre: Skipped 1 previous similar message [ 8899.560093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8899.567216] Lustre: Skipped 1 previous similar message [ 8899.791265] Lustre: server umount lustre-OST0000 complete [ 8904.676984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8904.680955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8904.703453] Lustre: Skipped 1 previous similar message [ 8904.709221] Lustre: Skipped 1 previous similar message [ 8910.027643] Lustre: server umount lustre-MDT0000 complete [ 8913.924614] LustreError: 305258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850247 with bad export cookie 7230724829328562181 [ 8913.931084] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8913.935017] LustreError: 305258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8914.183782] Lustre: server umount lustre-MDT0001 complete [ 8922.472621] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 13:04:14 (1787850254) [ 8924.160542] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8925.975785] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 13:04:18 (1787850258) [ 8929.397658] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 8953.978974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8954.178173] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8954.194632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8954.313899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8958.589850] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 8967.032384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8967.106298] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8967.161294] Lustre: 311093:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8971.502144] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 8992.112693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8992.290851] Lustre: Denying initial registration attempt from nid 192.168.206.105@tcp, specified as failover [ 8992.302806] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8992.312395] LustreError: 312545:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8992.320355] LustreError: 312545:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8992.324800] LustreError: 312545:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8992.383553] Lustre: server umount lustre-OST0000 complete [ 8992.389755] LustreError: 312545:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8999.905708] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8999.932900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8999.950498] Lustre: Skipped 1 previous similar message [ 8999.971430] Lustre: Skipped 3 previous similar messages [ 9013.215274] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9013.428894] Lustre: server umount lustre-MDT0000 complete [ 9015.267811] LustreError: 311100:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9015.281968] LustreError: 311100:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 9017.358422] LustreError: 311083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850351 with bad export cookie 7230724829328563063 [ 9017.379193] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9017.803260] Lustre: server umount lustre-MDT0001 complete [ 9030.666193] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 9033.267371] Key type lgssc unregistered [ 9033.679898] LNet: 313670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9033.686610] LNetError: 313670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9033.706685] LNet: Removed LNI 192.168.206.105@tcp [ 9034.485863] Key type .llcrypt unregistered [ 9034.488451] Key type ._llcrypt unregistered [ 9046.179172] Key type ._llcrypt registered [ 9046.183288] Key type .llcrypt registered [ 9046.269930] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 9047.265311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9047.297491] alg: No test for adler32 (adler32-zlib) [ 9048.389969] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 9048.641493] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 9050.303305] Key type lgssc registered [ 9051.301809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9095.308572] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 9103.464172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9103.478367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9104.653173] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9104.675380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9104.818604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9108.702152] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 9115.620224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9115.705996] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9115.742853] Lustre: 316435:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9115.931607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9119.135530] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 9126.246881] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9131.422986] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9150.205594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9150.460398] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9150.695653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9156.199233] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 9163.537990] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9167.841829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9167.856787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9167.881805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9168.358771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9168.363034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9171.873799] Lustre: server umount lustre-OST0000 complete [ 9177.568476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9177.580631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9177.596616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9177.604989] Lustre: Skipped 2 previous similar messages [ 9178.596638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9178.602924] Lustre: Skipped 1 previous similar message [ 9183.713503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9183.718615] Lustre: Skipped 1 previous similar message [ 9188.833757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9190.367854] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9190.606846] Lustre: server umount lustre-MDT0000 complete [ 9194.221967] LustreError: 318301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850528 with bad export cookie 7991640728278417171 [ 9194.224496] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9194.232385] LustreError: 318301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9194.711518] Lustre: server umount lustre-MDT0001 complete [ 9205.325170] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 9208.281636] Key type lgssc unregistered [ 9208.580851] LNet: 320011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9208.588764] LNetError: 320011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9208.606123] LNet: Removed LNI 192.168.206.105@tcp [ 9209.596137] Key type .llcrypt unregistered [ 9209.598546] Key type ._llcrypt unregistered [ 9237.491485] Key type ._llcrypt registered [ 9237.493297] Key type .llcrypt registered [ 9237.597936] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 9238.992162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9239.011108] alg: No test for adler32 (adler32-zlib) [ 9240.247565] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 9240.554101] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 9242.231155] Key type lgssc registered [ 9243.761593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9250.886200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9250.909960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9252.340733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9256.929098] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 9262.563204] LustreError: 321430:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9262.586460] LustreError: 321430:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9265.891493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9266.316448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9271.070275] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 9279.109264] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9284.465303] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9292.591836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9292.950221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9299.195374] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 9307.469702] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9322.464677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9322.469289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9322.478786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9322.990841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9323.005503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9323.009611] Lustre: Skipped 1 previous similar message [ 9325.644546] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.5@tcp (stopping) [ 9328.100851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9328.109645] Lustre: Skipped 2 previous similar messages [ 9333.229729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9333.240209] Lustre: Skipped 3 previous similar messages [ 9336.287365] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9336.643982] Lustre: server umount lustre-MDT0000 complete [ 9338.337784] LustreError: 321430:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9338.366592] LustreError: 321430:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9340.377422] LustreError: 321413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850674 with bad export cookie 12537961663894601707 [ 9340.383763] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9340.407376] LustreError: 321413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9340.968512] LustreError: 323921:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9343.458081] LustreError: 321425:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9343.459997] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9343.495412] Lustre: Skipped 1 previous similar message [ 9343.501382] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9343.508363] Lustre: Skipped 2 previous similar messages [ 9347.021372] Lustre: server umount lustre-MDT0001 complete [ 9353.496688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9362.146947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9366.421870] LustreError: 323260:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9366.441735] LustreError: 323260:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9366.484031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9366.532596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9370.894541] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 9375.596725] LustreError: 325046:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9375.624144] LustreError: 325046:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9375.721932] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9380.096982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9380.606991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9385.895152] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 9385.970299] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9394.482259] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9396.240830] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9402.562090] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9404.270111] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9411.557321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9411.558180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9411.576178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9411.589229] Lustre: Skipped 1 previous similar message [ 9411.595315] Lustre: Skipped 2 previous similar messages [ 9424.351133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9424.571324] Lustre: server umount lustre-OST0000 complete [ 9432.033715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9432.045675] Lustre: Skipped 1 previous similar message [ 9434.717540] Lustre: server umount lustre-MDT0000 complete [ 9437.157085] LustreError: 325050:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9437.182078] LustreError: 325050:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9438.501160] LustreError: 324238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850772 with bad export cookie 12537961663894609288 [ 9438.508048] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9438.516212] LustreError: 324238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9438.906995] Lustre: server umount lustre-MDT0001 complete [ 9450.856160] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 9454.435449] Key type lgssc unregistered [ 9454.809552] LNet: 327726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9454.817365] LNetError: 327726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9454.839407] LNet: Removed LNI 192.168.206.105@tcp [ 9455.963987] Key type .llcrypt unregistered [ 9455.967598] Key type ._llcrypt unregistered [ 9480.420235] Key type ._llcrypt registered [ 9480.422423] Key type .llcrypt registered [ 9480.508744] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 9481.531980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9481.622970] alg: No test for adler32 (adler32-zlib) [ 9482.787909] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 9483.213610] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 9484.895175] Key type lgssc registered [ 9486.259564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9495.061443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9495.113039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9496.515917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9503.686621] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 9506.792841] LustreError: 329142:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9506.829385] LustreError: 329142:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9511.917675] LustreError: 329141:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9517.029884] LustreError: 329142:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9517.878515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9518.692145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9524.338744] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 9532.700212] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9539.794404] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9544.162931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9544.179276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9544.199514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9548.775570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9548.778585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9550.433128] Lustre: server umount lustre-MDT0000 complete [ 9553.894591] LustreError: 329138:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9553.906442] LustreError: 329138:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9555.108729] LustreError: 329123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850888 with bad export cookie 14296988881798539646 [ 9555.122975] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9555.662606] Lustre: server umount lustre-MDT0001 complete [ 9570.687328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9571.133343] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9571.154554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9571.461637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9575.535846] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 9584.474649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9584.567707] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9584.606407] Lustre: 331732:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9584.877521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9589.101986] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 9596.224472] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9602.824384] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9612.397445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9612.624799] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9612.641091] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9612.650824] LustreError: 333617:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9612.664878] LustreError: 333617:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9612.671268] LustreError: 333617:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9612.781711] Lustre: server umount lustre-OST0000 complete [ 9612.783298] LustreError: 333617:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9623.205938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9623.410480] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9623.556939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9629.264988] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 9632.422800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9636.324395] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9643.906563] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9644.016727] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9644.099142] Lustre: lustre-OST0001: new disk, initializing [ 9644.108317] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9644.122456] Lustre: 335157:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9644.212901] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9649.907183] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 9651.324293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9651.335306] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9651.450873] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9656.570952] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9661.923810] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9661.926255] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9661.929717] LustreError: Skipped 1 previous similar message [ 9661.931336] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9661.944037] Lustre: Skipped 1 previous similar message [ 9661.945284] Lustre: Skipped 1 previous similar message [ 9665.119454] Lustre: server umount lustre-OST0001 complete [ 9667.044806] LustreError: 334085:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9667.060639] LustreError: 334085:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9668.065112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9668.077819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9668.096052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9668.100798] Lustre: Skipped 1 previous similar message [ 9672.164868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9672.171154] Lustre: Skipped 2 previous similar messages [ 9677.293624] LustreError: 334082:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9677.306592] LustreError: 334082:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9682.399127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9682.403076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9682.411013] Lustre: Skipped 3 previous similar messages [ 9682.550454] Lustre: server umount lustre-OST0000 complete [ 9687.523103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9687.532752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9687.533199] Lustre: Skipped 2 previous similar messages [ 9687.546650] LustreError: Skipped 1 previous similar message [ 9700.319151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9700.523471] Lustre: server umount lustre-MDT0000 complete [ 9702.884188] LustreError: 331738:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9702.896253] LustreError: 331738:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9704.075846] LustreError: 331724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851037 with bad export cookie 14296988881798540199 [ 9704.080856] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9704.084695] LustreError: 331724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9704.520945] Lustre: server umount lustre-MDT0001 complete [ 9715.598153] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 9718.176952] Key type lgssc unregistered [ 9718.457759] LNet: 337121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9718.467068] LNetError: 337121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9718.499101] LNet: Removed LNI 192.168.206.105@tcp [ 9719.310140] Key type .llcrypt unregistered [ 9719.312243] Key type ._llcrypt unregistered [ 9784.991796] Key type ._llcrypt registered [ 9784.994193] Key type .llcrypt registered [ 9785.088817] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 9798.499972] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 9799.521582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9799.732675] alg: No test for adler32 (adler32-zlib) [ 9800.738751] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 9801.080949] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 9802.759242] Key type lgssc registered [ 9803.976845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9833.315773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9833.361685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9834.596782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9834.634405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9834.703518] Lustre: lustre-MDT0000: new disk, initializing [ 9834.752426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9834.776927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9838.519312] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 9846.183079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9846.259454] Lustre: 342042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9846.300971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9846.304582] Lustre: Skipped 1 previous similar message [ 9846.357873] Lustre: lustre-MDT0001: new disk, initializing [ 9846.417056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9846.442793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9846.450510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9850.271309] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 9858.469854] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9863.179229] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9868.453647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9868.713340] Lustre: lustre-OST0000: new disk, initializing [ 9868.720424] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9868.727926] Lustre: 344012:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9868.808516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9874.103135] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 9878.627471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9878.647244] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9878.760465] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9882.022900] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9886.212488] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9886.391979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9889.903911] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9890.047536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9894.370636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9894.372382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9894.386809] LustreError: Skipped 1 previous similar message [ 9894.400492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9894.406954] Lustre: Skipped 1 previous similar message [ 9897.855852] Lustre: server umount lustre-OST0000 complete [ 9903.073143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9903.082415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9903.103041] Lustre: Skipped 1 previous similar message [ 9903.109404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9904.608333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9904.613058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9908.064809] Lustre: server umount lustre-MDT0000 complete [ 9912.167725] LustreError: 342848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851245 with bad export cookie 12356899552895297015 [ 9912.169229] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9912.189858] LustreError: 342848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9913.044891] Lustre: server umount lustre-MDT0001 complete [ 9922.919263] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 13:20:54 (1787851254) [ 9933.538840] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 13:21:04 (1787851264) [ 9943.490927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9944.027838] LustreError: 346766:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9944.049430] LustreError: 346766:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9944.103768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9949.155858] LustreError: 346767:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9949.407553] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 9953.259086] LustreError: 346766:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9958.379979] LustreError: 346767:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9958.836471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9959.077332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9964.195834] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 9972.182551] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9977.580659] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9984.888799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9985.350619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9991.448823] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 9999.764358] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10010.164671] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10011.913171] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10013.430973] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10015.073731] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10017.296633] Lustre: Failing over lustre-MDT0000 [10017.646864] Lustre: server umount lustre-MDT0000 complete [10019.302958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10019.329423] Lustre: Skipped 2 previous similar messages [10019.333810] LustreError: 346763:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10019.353788] LustreError: 346763:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10029.541705] LustreError: 346762:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10029.568094] LustreError: 346762:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [10035.081350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10035.169811] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10035.361920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10035.434482] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10037.288695] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10039.762491] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [10040.832561] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10040.877195] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10040.997563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10048.446640] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10049.653967] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10053.016993] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10054.861268] Lustre: Failing over lustre-MDT0000 [10055.213331] Lustre: server umount lustre-MDT0000 complete [10056.167147] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10056.174603] LustreError: 347495:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10056.174856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10056.182083] Lustre: Skipped 2 previous similar messages [10056.212924] LustreError: 347495:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [10071.519339] Lustre: 340269:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851389/real 1787851389] req@000000007b01bc0e x1874697790769920/t0(0) o400->MGC192.168.206.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787851405 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10071.543812] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10074.571768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10081.767623] LustreError: 340268:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000af9139a x1874697790775552/t0(0) o250->MGC192.168.206.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10082.073378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10082.115701] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10084.420505] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10086.518428] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [10087.429427] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10087.441136] Lustre: Skipped 2 previous similar messages [10087.484438] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10087.556976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10095.463146] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10096.955802] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10102.759225] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10102.764480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10102.771636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10102.779507] Lustre: Skipped 2 previous similar messages [10102.795919] Lustre: Skipped 3 previous similar messages [10107.875502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10107.890660] Lustre: Skipped 2 previous similar messages [10108.975933] Lustre: server umount lustre-MDT0000 complete [10113.002185] LustreError: 346767:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10113.022645] LustreError: 346767:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 23 previous similar messages [10113.193348] LustreError: 346748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851447 with bad export cookie 12356899552895299724 [10113.205612] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10113.533639] Lustre: server umount lustre-MDT0001 complete [10128.236853] Lustre: server umount lustre-OST0000 complete [10144.008000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10150.624167] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [10159.880990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10160.250617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10160.264358] Lustre: Skipped 1 previous similar message [10164.244668] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [10171.608672] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10177.874511] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10186.361899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10189.811240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10193.780215] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [10201.443266] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10214.024362] Lustre: server umount lustre-OST0000 complete [10221.536459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10221.541883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10221.557177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10221.566129] Lustre: Skipped 1 previous similar message [10221.584896] Lustre: Skipped 1 previous similar message [10227.755833] Lustre: server umount lustre-MDT0000 complete [10231.315660] LustreError: 353087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851565 with bad export cookie 12356899552895300473 [10231.332676] LustreError: 353087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10231.332773] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10241.751730] Lustre: server umount lustre-MDT0001 complete [10252.678576] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [10255.828553] Key type lgssc unregistered [10256.092596] LNet: 356644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10256.098989] LNetError: 356644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10256.111266] LNet: Removed LNI 192.168.206.105@tcp [10256.935334] Key type .llcrypt unregistered [10256.936978] Key type ._llcrypt unregistered [10284.723981] Key type ._llcrypt registered [10284.727513] Key type .llcrypt registered [10284.833205] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [10286.178773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10286.238553] alg: No test for adler32 (adler32-zlib) [10287.243879] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [10287.488059] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [10289.207169] Key type lgssc registered [10290.434642] Lustre: Echo OBD driver; http://www.lustre.org/ [10298.504181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10298.516198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10300.314046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10304.852410] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [10310.628389] LustreError: 358162:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10310.664364] LustreError: 358162:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10314.479900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10314.850835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10320.049738] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [10327.790823] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10334.147221] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10343.669747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10344.462270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10345.510049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10354.260406] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [10364.824315] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10376.173283] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10376.182192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10376.205340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10380.776413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10380.779824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10380.787973] Lustre: Skipped 1 previous similar message [10380.800065] Lustre: Skipped 2 previous similar messages [10382.004409] Lustre: server umount lustre-MDT0000 complete [10385.892145] LustreError: 358161:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10385.926964] LustreError: 358161:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10387.535795] LustreError: 360657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851721 with bad export cookie 3655770505808779136 [10387.549379] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10387.557792] LustreError: 360657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10388.266823] Lustre: server umount lustre-MDT0001 complete [10405.353473] Lustre: server umount lustre-OST0000 complete [10421.845343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10422.491722] LustreError: 361879:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10422.599420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10427.794618] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [10427.874679] LustreError: 361880:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10433.006971] LustreError: 361879:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10436.965578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10437.459340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10442.430556] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [10451.295949] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10457.804968] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10465.061135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10465.404849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10467.448080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10471.266186] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [10479.821658] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10493.922485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10493.940076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10493.954542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10498.322500] Lustre: server umount lustre-MDT0000 complete [10499.556281] LustreError: 361874:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10499.568286] LustreError: 361874:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10501.716896] LustreError: 361859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851835 with bad export cookie 3655770505808780249 [10501.727198] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10502.080315] Lustre: server umount lustre-MDT0001 complete [10516.259994] Lustre: server umount lustre-OST0000 complete [10534.497479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10534.932965] LustreError: 365789:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10535.022235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10539.760566] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [10549.449395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10549.730949] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10549.746933] Lustre: Skipped 3 previous similar messages [10554.986649] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [10563.361967] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10568.905904] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10575.588993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10575.877535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10575.883132] Lustre: Skipped 1 previous similar message [10580.013721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10581.641743] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [10590.246317] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10599.396219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10599.411211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10599.411710] Lustre: Skipped 2 previous similar messages [10599.412786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10602.319937] Lustre: server umount lustre-OST0000 complete [10609.645810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10609.651907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10609.664244] Lustre: Skipped 2 previous similar messages [10609.686264] Lustre: Skipped 2 previous similar messages [10612.995894] Lustre: server umount lustre-MDT0000 complete [10614.753523] LustreError: 365786:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10614.766405] LustreError: 365786:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [10616.997667] LustreError: 365770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851950 with bad export cookie 3655770505808781271 [10617.003830] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10617.009075] LustreError: 365770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10617.319895] Lustre: server umount lustre-MDT0001 complete [10630.401071] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [10634.270685] Key type lgssc unregistered [10634.687398] LNet: 369325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10634.697807] LNetError: 369325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10635.761249] LNet: Removed LNI 192.168.206.105@tcp [10637.227905] Key type .llcrypt unregistered [10637.232459] Key type ._llcrypt unregistered [10667.014454] Key type ._llcrypt registered [10667.022866] Key type .llcrypt registered [10667.165566] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [10667.919342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10667.983462] alg: No test for adler32 (adler32-zlib) [10669.383328] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [10669.725634] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [10671.607169] Key type lgssc registered [10672.554124] Lustre: Echo OBD driver; http://www.lustre.org/ [10681.124828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10681.139074] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10681.146412] LustreError: 370770:0:(osd_handler.c:8517:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10681.166798] LustreError: 370770:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10681.171212] LustreError: 370770:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10681.177412] LustreError: 370770:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10681.183408] LustreError: 370770:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10690.116314] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10690.125916] LustreError: 371167:0:(osd_handler.c:8517:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10690.141085] LustreError: 371167:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10690.146612] LustreError: 371167:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10690.152597] LustreError: 371167:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10690.163736] LustreError: 371167:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10714.305658] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [10717.236666] Key type lgssc unregistered [10717.487073] LNet: 372174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10717.506749] LNetError: 372174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10717.524858] LNet: Removed LNI 192.168.206.105@tcp [10718.326899] Key type .llcrypt unregistered [10718.329125] Key type ._llcrypt unregistered [10736.886337] Key type ._llcrypt registered [10736.892369] Key type .llcrypt registered [10737.037155] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [10751.272143] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [10752.200138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10752.214379] alg: No test for adler32 (adler32-zlib) [10753.187692] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [10753.399308] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [10755.079333] Key type lgssc registered [10756.156922] Lustre: Echo OBD driver; http://www.lustre.org/ [10797.110155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10797.137172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10798.402354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10798.435794] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10798.551056] Lustre: lustre-MDT0000: new disk, initializing [10798.610833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10798.624327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10804.124765] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [10818.251863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10818.497227] Lustre: 375332:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10818.630839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10818.637128] Lustre: Skipped 1 previous similar message [10818.823866] Lustre: lustre-MDT0001: new disk, initializing [10819.080489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10819.130749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10819.148190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10827.293686] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [10840.080492] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10848.884725] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10856.953473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10857.265428] Lustre: lustre-OST0000: new disk, initializing [10857.275878] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10857.282429] Lustre: 377303:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10857.346193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10864.014117] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [10866.755151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10866.764938] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10866.879066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10874.140731] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10878.407434] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10878.746760] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10882.739429] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10882.977052] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10887.652314] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10887.652411] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10887.663708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10887.686757] Lustre: Skipped 1 previous similar message [10887.707434] Lustre: Skipped 1 previous similar message [10891.300298] Lustre: server umount lustre-OST0000 complete [10895.850522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10895.866789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10895.895913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10897.890777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10897.892508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10901.661399] Lustre: server umount lustre-MDT0000 complete [10905.910032] LustreError: 375324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852239 with bad export cookie 2788945433858109334 [10905.930611] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10905.942083] LustreError: 375324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10906.573730] Lustre: server umount lustre-MDT0001 complete [10917.599413] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10919.635405] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:37:31 (1787852251) [10927.183451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10927.486891] LustreError: 379761:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10927.506465] LustreError: 379761:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10927.549836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10932.651512] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [10932.711941] LustreError: 379762:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10937.828838] LustreError: 379761:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10942.950090] LustreError: 379762:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10943.204676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10943.714832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10948.584337] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [10956.700774] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10962.980528] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10969.738714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10970.139982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10976.317172] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [10984.163975] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10991.512564] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10991.690803] Lustre: lustre-OST0001: new disk, initializing [10991.701340] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10991.712799] Lustre: 382602:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10991.846193] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10998.497164] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [10998.876392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10998.889920] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10999.014092] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11007.668719] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11014.624789] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11014.625699] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11014.644866] LustreError: Skipped 1 previous similar message [11014.655936] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11014.657526] Lustre: Skipped 1 previous similar message [11014.676222] Lustre: Skipped 2 previous similar messages [11018.563198] Lustre: server umount lustre-OST0001 complete [11019.748936] LustreError: 381585:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11019.785643] LustreError: 381585:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11030.019814] LustreError: 381583:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11030.052769] LustreError: 381583:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [11038.199477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11038.211297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11038.214945] Lustre: Skipped 1 previous similar message [11038.227351] Lustre: Skipped 2 previous similar messages [11041.854710] Lustre: server umount lustre-OST0000 complete [11048.419733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11048.436474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11048.441078] Lustre: Skipped 1 previous similar message [11048.458224] Lustre: Skipped 1 previous similar message [11053.387188] Lustre: server umount lustre-MDT0000 complete [11053.545990] LustreError: 380491:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11053.583752] LustreError: 380491:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [11058.985259] LustreError: 379741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852392 with bad export cookie 2788945433858110216 [11058.990764] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11058.994470] LustreError: 379741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11065.473144] Lustre: server umount lustre-MDT0001 complete [11079.272868] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [11082.488669] Key type lgssc unregistered [11082.832321] LNet: 384592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11082.851833] LNetError: 384592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11083.885327] LNet: Removed LNI 192.168.206.105@tcp [11085.084599] Key type .llcrypt unregistered [11085.088712] Key type ._llcrypt unregistered [11141.023681] Key type ._llcrypt registered [11141.026608] Key type .llcrypt registered [11141.156958] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [11142.849158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11142.890776] alg: No test for adler32 (adler32-zlib) [11144.038118] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [11144.361509] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [11146.055763] Key type lgssc registered [11147.635929] Lustre: Echo OBD driver; http://www.lustre.org/ [11156.585643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11156.604065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11157.832461] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11157.848434] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11158.009081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11162.326440] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [11172.729128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11172.927789] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11172.974764] Lustre: 387391:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11173.434982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11180.138723] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [11191.896257] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11199.841656] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11204.064638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11204.069117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11204.085434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11206.114945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11206.116642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11206.138195] Lustre: Skipped 1 previous similar message [11211.234089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11211.241813] Lustre: Skipped 1 previous similar message [11216.352194] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11216.355079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11216.372123] Lustre: Skipped 1 previous similar message [11216.625890] Lustre: server umount lustre-MDT0000 complete [11220.642360] LustreError: 387384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852554 with bad export cookie 16684135206443635673 [11220.657116] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11221.130365] Lustre: server umount lustre-MDT0001 complete [11235.466291] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:42:46 (1787852566) [11245.361744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11246.053374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11251.171950] LustreError: 389908:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11251.202093] LustreError: 389908:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11251.262116] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [11256.290557] LustreError: 389908:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11260.158858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11260.409480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11264.422878] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [11272.251526] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11276.967370] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11284.492969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11284.667155] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11284.859872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11292.440674] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [11302.561795] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11311.656198] Lustre: Permanently deactivating lustre-OST0000 [11311.711473] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11318.315618] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11320.762681] LustreError: 392850:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11320.778942] LustreError: 392850:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11320.784967] LustreError: 392850:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11323.941529] LustreError: 392899:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11323.953993] LustreError: 392899:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11332.335272] Lustre: server umount lustre-OST0000 complete [11336.229283] LustreError: 393149:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11336.233573] LustreError: 393149:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11341.794319] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11341.803330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11341.806151] Lustre: Skipped 1 previous similar message [11341.829181] Lustre: Skipped 1 previous similar message [11349.063415] Lustre: server umount lustre-MDT0000 complete [11352.034198] LustreError: 389908:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11352.082617] LustreError: 389908:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11353.484228] LustreError: 389887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852687 with bad export cookie 16684135206443636254 [11353.488110] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11353.501140] LustreError: 389887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11364.025693] Lustre: server umount lustre-MDT0001 complete [11376.270896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11381.660556] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [11384.276895] LustreError: 394309:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11390.956306] Lustre: 394648:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.105@tcp [11393.216175] Lustre: server umount MGS complete [11404.689061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11405.118341] LustreError: 395373:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11405.170828] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11405.178356] Lustre: Skipped 1 previous similar message [11405.228586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11409.807635] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [11410.403939] LustreError: 395374:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11417.622785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11417.837101] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11421.711764] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [11428.866494] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11435.081717] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11437.467157] Lustre: Permanently reactivating lustre-OST0000 [11437.489593] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11437.497225] Lustre: Skipped 2 previous similar messages [11466.721444] LustreError: 397175:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11474.869561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11475.218171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11475.223149] Lustre: Skipped 1 previous similar message [11481.437340] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [11490.417972] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11502.048301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11502.057317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11502.087791] Lustre: Skipped 1 previous similar message [11502.089659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11502.113822] Lustre: Skipped 3 previous similar messages [11511.263151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11511.431838] Lustre: server umount lustre-OST0000 complete [11517.413499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11517.436643] Lustre: Skipped 1 previous similar message [11522.313926] Lustre: server umount lustre-MDT0000 complete [11522.528830] LustreError: 396807:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11522.544179] LustreError: 396807:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11526.372584] LustreError: 396110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852860 with bad export cookie 16684135206443637304 [11526.380421] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11526.386881] LustreError: 396110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11527.116222] Lustre: server umount lustre-MDT0001 complete [11539.052533] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [11541.545968] Key type lgssc unregistered [11541.817558] LNet: 399790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11541.823968] LNetError: 399790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11541.841984] LNet: Removed LNI 192.168.206.105@tcp [11542.674785] Key type .llcrypt unregistered [11542.679244] Key type ._llcrypt unregistered [11563.084474] Key type ._llcrypt registered [11563.087341] Key type .llcrypt registered [11563.202673] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [11578.501389] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [11579.503355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11579.537276] alg: No test for adler32 (adler32-zlib) [11580.559892] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [11580.745655] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [11582.385279] Key type lgssc registered [11583.315430] Lustre: Echo OBD driver; http://www.lustre.org/ [11616.107826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11616.135099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11617.403872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11617.434605] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11617.530793] Lustre: lustre-MDT0000: new disk, initializing [11617.624143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11617.641943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11622.226405] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [11631.104755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11631.244873] Lustre: 402947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11631.312535] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11631.325869] Lustre: Skipped 1 previous similar message [11631.432382] Lustre: lustre-MDT0001: new disk, initializing [11631.508397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11631.537505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11631.544991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11635.903491] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [11644.599421] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11649.357119] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11657.569458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11657.897497] Lustre: lustre-OST0000: new disk, initializing [11657.902719] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11657.911526] Lustre: 404913:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11658.018560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11664.734779] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [11665.519161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11665.534446] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11665.658618] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11673.794373] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11677.645384] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11677.806297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11682.197674] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11682.511414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11686.376654] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11686.379060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11686.384161] LustreError: Skipped 1 previous similar message [11686.390261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11686.406699] Lustre: Skipped 1 previous similar message [11690.810969] Lustre: server umount lustre-OST0000 complete [11696.614846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11696.621877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11696.636053] Lustre: Skipped 1 previous similar message [11696.651389] Lustre: Skipped 2 previous similar messages [11700.664765] Lustre: server umount lustre-MDT0000 complete [11704.439296] LustreError: 402938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853038 with bad export cookie 4296043064002982412 [11704.440961] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11704.447916] LustreError: 402938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11704.717817] Lustre: server umount lustre-MDT0001 complete [11713.535648] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11715.568276] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:50:47 (1787853047) [11724.803048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11725.330322] LustreError: 407378:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11725.344784] LustreError: 407378:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11725.391625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11729.526981] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [11730.400918] LustreError: 407379:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11735.523599] LustreError: 407378:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11739.153393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11739.604906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11744.198464] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [11750.444593] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11755.604567] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11761.103269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11761.457091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11766.859982] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [11774.046944] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11775.642433] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11778.206702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11785.184774] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11785.212089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11785.239769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11786.211489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11789.737454] Lustre: server umount lustre-OST0000 complete [11795.942873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11795.953163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11795.966751] Lustre: Skipped 1 previous similar message [11795.970802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11795.974681] Lustre: Skipped 2 previous similar messages [11800.638128] Lustre: server umount lustre-MDT0000 complete [11801.571247] LustreError: 407378:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11801.606035] LustreError: 407378:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11805.238229] LustreError: 407360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853139 with bad export cookie 4296043064002983294 [11805.245561] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11805.247055] LustreError: 407360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11805.585047] Lustre: server umount lustre-MDT0001 complete [11815.916857] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [11818.363933] Key type lgssc unregistered [11818.732843] LNet: 411106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11818.754166] LNetError: 411106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11818.775485] LNet: Removed LNI 192.168.206.105@tcp [11819.873752] Key type .llcrypt unregistered [11819.875406] Key type ._llcrypt unregistered [11843.741706] Key type ._llcrypt registered [11843.744363] Key type .llcrypt registered [11843.824751] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [11844.091390] Key type .llcrypt unregistered [11844.095444] Key type ._llcrypt unregistered [11859.503146] Key type ._llcrypt registered [11859.504683] Key type .llcrypt registered [11859.677707] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [11860.912744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11860.938632] alg: No test for adler32 (adler32-zlib) [11861.947795] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [11862.138610] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [11863.808316] Key type lgssc registered [11864.804611] Lustre: Echo OBD driver; http://www.lustre.org/ [11872.128135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11872.142782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11873.635542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11878.133705] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [11884.009434] LustreError: 413329:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11884.041308] LustreError: 413329:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11886.720214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11887.070746] LustreError: 413328:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11887.087633] LustreError: 413328:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11887.105146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11892.192712] LustreError: 413329:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11892.218977] LustreError: 413329:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11893.404295] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [11897.316925] LustreError: 413328:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11897.354122] LustreError: 413328:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11898.359421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11901.257653] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11903.467271] LustreError: 413329:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11903.483099] LustreError: 413329:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11908.221296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11908.760701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11913.313416] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [11923.585751] Lustre: server umount lustre-OST0000 complete [11929.064103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11929.069363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11929.073209] LustreError: Skipped 1 previous similar message [11929.118855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11934.690416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11934.703972] Lustre: Skipped 2 previous similar messages [11937.860659] Lustre: server umount lustre-MDT0000 complete [11939.809130] Lustre: 412642:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787853257/real 1787853257] req@00000000ab3c1ac7 x1874699951826944/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787853273 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11939.812365] LustreError: 414604:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11939.837415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11939.837423] Lustre: Skipped 1 previous similar message [11939.867608] LustreError: 414604:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11940.721112] Lustre: 412640:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787853258/real 1787853258] req@000000007f082e45 x1874699951827712/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787853274 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11941.674042] LustreError: 413310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853275 with bad export cookie 7656785648486796006 [11941.679769] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11941.684836] LustreError: 413310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11942.091407] Lustre: server umount lustre-MDT0001 complete [11952.551811] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [11955.201940] Key type lgssc unregistered [11955.681714] LNet: 416480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11955.686991] LNetError: 416480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11955.698176] LNet: Removed LNI 192.168.206.105@tcp [11956.504418] Key type .llcrypt unregistered [11956.512203] Key type ._llcrypt unregistered [11982.872837] Key type ._llcrypt registered [11982.874801] Key type .llcrypt registered [11982.974585] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [11984.024688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11984.061462] alg: No test for adler32 (adler32-zlib) [11985.303716] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [11985.726610] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [11987.457715] Key type lgssc registered [11989.236279] Lustre: Echo OBD driver; http://www.lustre.org/ [12000.268744] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12000.293839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12015.903985] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12021.023375] LustreError: 417873:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.105@tcp: failed processing log, type 4: rc = -110 [12047.712543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12053.525491] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [12060.611919] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12067.959048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12068.421633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12068.478212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12072.499595] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [12073.443247] LustreError: 417898:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12073.461930] LustreError: 417898:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12078.581771] LustreError: 418934:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12080.406172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12080.849804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12085.445183] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [12092.189748] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12098.493624] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12106.720834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12106.737844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12106.767305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12107.814284] Lustre: server umount lustre-OST0000 complete [12112.869931] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12112.881413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12112.901476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12113.900225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12113.905591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12113.937115] Lustre: Skipped 1 previous similar message [12119.011994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12119.027138] Lustre: Skipped 1 previous similar message [12122.187062] Lustre: server umount lustre-MDT0000 complete [12124.129712] LustreError: 418917:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12124.156982] LustreError: 418917:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12125.902826] LustreError: 417879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853459 with bad export cookie 4192070267540240332 [12125.918151] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12125.925686] LustreError: 417879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12126.240941] Lustre: server umount lustre-MDT0001 complete [12136.898788] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [12139.620336] Key type lgssc unregistered [12139.913632] LNet: 421447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12139.932191] LNetError: 421447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12140.968370] LNet: Removed LNI 192.168.206.105@tcp [12141.880424] Key type .llcrypt unregistered [12141.882873] Key type ._llcrypt unregistered [12167.604619] Key type ._llcrypt registered [12167.609269] Key type .llcrypt registered [12167.735578] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [12168.406269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12168.428242] alg: No test for adler32 (adler32-zlib) [12169.616191] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [12169.917040] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [12171.599163] Key type lgssc registered [12172.871381] Lustre: Echo OBD driver; http://www.lustre.org/ [12181.139872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12181.168043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12182.632235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12187.773707] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [12192.248179] LustreError: 422867:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12192.276089] LustreError: 422867:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12195.991092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12196.534428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12201.203422] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [12208.970958] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12215.159275] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12221.940062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12222.280927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12227.381547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12228.210718] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [12235.162740] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12241.892854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12241.903575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12241.913832] Lustre: Skipped 1 previous similar message [12241.925210] Lustre: Skipped 2 previous similar messages [12243.035803] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.5@tcp (stopping) [12246.161338] Lustre: server umount lustre-MDT0000 complete [12247.020680] LustreError: 422866:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12247.039032] LustreError: 422866:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12248.403833] LustreError: 422845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.5@tcp arrived at 1787853582 with bad export cookie 11370531930296944608 [12248.417611] LustreError: 422845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12250.346078] LustreError: 422846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853584 with bad export cookie 11370531930296943719 [12250.367031] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12252.128881] LustreError: 422867:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12252.131624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12252.143415] LustreError: 422867:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12252.171803] Lustre: Skipped 1 previous similar message [12252.176878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12255.137959] LustreError: 423594:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12255.138977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12255.151234] LustreError: 423594:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12256.559515] Lustre: server umount lustre-OST0000 complete [12260.263351] LustreError: 422863:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12260.281441] LustreError: 422863:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12262.391280] LustreError: 423596:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000074ed83c4 x1874700274793088/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12262.586491] Lustre: server umount lustre-MDT0001 complete [12272.342467] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [12275.463865] Key type lgssc unregistered [12275.791698] LNet: 426552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12275.797424] LNetError: 426552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12275.825399] LNet: Removed LNI 192.168.206.105@tcp [12276.698983] Key type .llcrypt unregistered [12276.700722] Key type ._llcrypt unregistered [12298.046703] Key type ._llcrypt registered [12298.048602] Key type .llcrypt registered [12298.157799] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [12299.017825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12299.069489] alg: No test for adler32 (adler32-zlib) [12300.171706] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [12300.405313] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [12302.055149] Key type lgssc registered [12303.088414] Lustre: Echo OBD driver; http://www.lustre.org/ [12310.561302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12310.594836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12311.986373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12316.804747] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [12321.272965] LustreError: 427947:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12321.291561] LustreError: 427947:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12326.372578] LustreError: 427948:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12326.683989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12327.043906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12331.534290] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [12338.908680] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12344.813354] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12351.665060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12351.970195] LustreError: 429780:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12352.111442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12356.234171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12358.500671] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [12366.025354] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12371.939268] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12371.955377] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12371.964243] Lustre: Skipped 1 previous similar message [12374.092762] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.5@tcp (stopping) [12376.981640] Lustre: server umount lustre-MDT0001 complete [12377.066325] LustreError: 427948:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12377.086671] LustreError: 427948:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12382.190378] LustreError: 430279:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12382.226211] LustreError: 430279:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12387.297370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12387.311676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12387.320525] Lustre: Skipped 1 previous similar message [12387.327404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12389.366689] Lustre: server umount lustre-OST0000 complete [12392.419206] LustreError: 427948:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12392.440820] LustreError: 427948:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12393.263760] Lustre: server umount lustre-MDT0000 complete [12405.345168] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [12407.766353] Key type lgssc unregistered [12408.048430] LNet: 431649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12408.059913] LNetError: 431649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12408.077991] LNet: Removed LNI 192.168.206.105@tcp [12408.754264] Key type .llcrypt unregistered [12408.756756] Key type ._llcrypt unregistered [12431.465391] Key type ._llcrypt registered [12431.467318] Key type .llcrypt registered [12431.559754] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [12445.549230] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [12446.557623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12446.590082] alg: No test for adler32 (adler32-zlib) [12447.732741] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [12448.007237] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [12449.671159] Key type lgssc registered [12450.631814] Lustre: Echo OBD driver; http://www.lustre.org/ [12484.711495] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [12500.496095] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12500.522366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12501.906683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12501.958207] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12502.114155] Lustre: lustre-MDT0000: new disk, initializing [12502.210747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12502.237694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12507.444749] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [12525.970133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12526.352812] Lustre: lustre-OST0000: new disk, initializing [12526.361827] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12526.364359] Lustre: Skipped 1 previous similar message [12526.376387] Lustre: 436684:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12526.445403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12532.847095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12532.856639] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12532.888667] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12533.605654] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [12543.574527] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12559.897023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12560.074800] Lustre: 435665:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12560.244430] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12560.382674] Lustre: lustre-MDT0001: new disk, initializing [12560.595607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12560.686853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12560.703647] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12566.365190] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [12577.682225] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12577.846957] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12583.212993] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12583.479862] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12591.087287] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12591.108672] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12591.119440] Lustre: Skipped 1 previous similar message [12591.673532] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.5@tcp (stopping) [12594.988981] Lustre: server umount lustre-MDT0001 complete [12601.317420] LustreError: 435675:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12601.339364] LustreError: 435675:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [12606.434569] LustreError: 435670:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12606.450968] LustreError: 435670:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12611.560513] LustreError: 435671:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12611.586235] LustreError: 435671:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12614.624648] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12614.641746] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12614.659555] Lustre: Skipped 1 previous similar message [12614.672097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12616.681894] LustreError: 435674:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12616.683384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12619.711739] Lustre: server umount lustre-OST0000 complete [12621.801343] LustreError: 435675:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12629.730157] Lustre: server umount lustre-MDT0000 complete [12644.599518] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [12647.479651] Key type lgssc unregistered [12647.841788] LNet: 440433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12647.853233] LNetError: 440433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12647.884124] LNet: Removed LNI 192.168.206.105@tcp [12648.792171] Key type .llcrypt unregistered [12648.794461] Key type ._llcrypt unregistered [12691.026716] Key type ._llcrypt registered [12691.031335] Key type .llcrypt registered [12691.217330] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [12691.573360] Key type .llcrypt unregistered [12691.576355] Key type ._llcrypt unregistered [12709.632092] Key type ._llcrypt registered [12709.637058] Key type .llcrypt registered [12709.729243] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [12710.935018] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12711.072073] alg: No test for adler32 (adler32-zlib) [12712.221672] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [12712.436199] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [12714.143924] Key type lgssc registered [12715.487434] Lustre: Echo OBD driver; http://www.lustre.org/ [12747.155282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12747.185222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12748.417452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12748.459508] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12748.648721] Lustre: lustre-MDT0000: new disk, initializing [12748.750216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12748.780969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12753.877958] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [12768.902170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12769.017437] Lustre: 443817:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12769.078630] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12769.082042] Lustre: Skipped 1 previous similar message [12769.147425] Lustre: lustre-MDT0001: new disk, initializing [12769.355729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12769.429053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12769.451880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12776.129921] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [12787.185354] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12793.649961] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12800.919484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12801.382415] Lustre: lustre-OST0000: new disk, initializing [12801.388283] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12801.420520] Lustre: 445783:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12801.584536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12806.778380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12806.794333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12806.930937] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12809.095917] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [12819.191803] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12827.618024] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12827.619987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12827.637935] Lustre: Skipped 2 previous similar messages [12827.653458] Lustre: Skipped 2 previous similar messages [12832.749403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12832.760396] Lustre: Skipped 2 previous similar messages [12833.270548] Lustre: server umount lustre-MDT0000 complete [12837.143454] LustreError: 444610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854170 with bad export cookie 4151456085390215700 [12837.147713] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12837.168066] LustreError: 444610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12837.575740] Lustre: server umount lustre-MDT0001 complete [12851.554303] Lustre: server umount lustre-OST0000 complete [12862.863821] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 14:09:55 (1787854195) [12871.652526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12872.096807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12877.181804] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [12877.280850] LustreError: 447814:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12877.308433] LustreError: 447814:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12881.381791] LustreError: 447815:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12886.506504] LustreError: 447814:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12886.582606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12886.884264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12891.796611] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [12899.867038] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12905.415411] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12926.062203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12926.313635] Lustre: Found index 0 for lustre-OST0000, updating log [12926.508711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12933.607919] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [12942.371239] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12942.903664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12953.056482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12953.065079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12953.079890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12953.128763] Lustre: Skipped 1 previous similar message [12958.899500] Lustre: server umount lustre-OST0000 complete [12974.688445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12974.706848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12974.713218] Lustre: Skipped 3 previous similar messages [12979.180103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12979.194870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12979.223544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12981.128792] Lustre: server umount lustre-MDT0000 complete [12981.217167] LustreError: 447814:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12981.235747] LustreError: 447814:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12984.972612] LustreError: 447795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854318 with bad export cookie 4151456085390217093 [12984.982291] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12984.987915] LustreError: 447795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12985.290078] Lustre: server umount lustre-MDT0001 complete [12988.959340] LNet: 1 local NIs in recovery (showing 1): 192.168.206.105@tcp [12994.581497] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 14:12:06 (1787854326) [13614.995728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13615.501545] LustreError: 454228:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13615.520585] LustreError: 454228:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13615.576308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13619.967349] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [13620.706686] LustreError: 454229:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13625.825790] LustreError: 454228:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13627.039241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13627.411116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13631.939300] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [13639.032717] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13643.968481] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13662.782115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13663.038606] Lustre: Found index 0 for lustre-OST0000, updating log [13663.427027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13664.435347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [13666.848097] LNet: 1 local NIs in recovery (showing 1): 192.168.206.105@tcp [13669.269333] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [13676.119163] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13678.947991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13678.966947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13678.977691] Lustre: Skipped 2 previous similar messages [13692.386995] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13692.530766] Lustre: server umount lustre-OST0000 complete [13697.190686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13697.208108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13697.217282] Lustre: Skipped 1 previous similar message [13699.299119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13699.315483] Lustre: Skipped 3 previous similar messages [13702.369926] LustreError: 454229:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13702.385966] LustreError: 454229:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13702.429543] Lustre: server umount lustre-MDT0000 complete [13705.656491] LustreError: 454208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855039 with bad export cookie 4151456085390218178 [13705.657804] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13705.667098] LustreError: 454208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13705.989460] Lustre: server umount lustre-MDT0001 complete [13726.239223] LNet: 1 local NIs in recovery (showing 1): 192.168.206.105@tcp [13737.267599] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [13746.786773] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [13785.057081] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 14:25:17 (1787855117) [13801.004978] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [13804.256512] Key type lgssc unregistered [13804.548634] LNet: 460853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13804.556944] LNetError: 460853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13923.384116] LNet: Removed LNI 192.168.206.105@tcp [13924.669228] Key type .llcrypt unregistered [13924.673776] Key type ._llcrypt unregistered [13941.127378] Key type ._llcrypt registered [13941.130561] Key type .llcrypt registered [13941.240881] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [13942.296033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13942.362702] alg: No test for adler32 (adler32-zlib) [13943.423910] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [13943.654968] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [13945.327354] Key type lgssc registered [13946.454951] Lustre: Echo OBD driver; http://www.lustre.org/ [13950.822744] LNet: Added LNI 192.168.206.105@tcp100 [8/256/0/180] [13951.591816] LNet: Added LNI 192.168.206.105@tcp101 [8/256/0/180] [13952.340499] LNet: Added LNI 192.168.206.105@tcp102 [8/256/0/180] [13953.087342] LNet: Added LNI 192.168.206.105@tcp103 [8/256/0/180] [13953.905549] LNet: Added LNI 192.168.206.105@tcp104 [8/256/0/180] [13954.664377] LNet: Added LNI 192.168.206.105@tcp105 [8/256/0/180] [13955.386876] LNet: Added LNI 192.168.206.105@tcp106 [8/256/0/180] [13956.121378] LNet: Added LNI 192.168.206.105@tcp107 [8/256/0/180] [13956.841288] LNet: Added LNI 192.168.206.105@tcp108 [8/256/0/180] [13957.678362] LNet: Added LNI 192.168.206.105@tcp109 [8/256/0/180] [13958.602460] LNet: Added LNI 192.168.206.105@tcp110 [8/256/0/180] [13959.371619] LNet: Added LNI 192.168.206.105@tcp111 [8/256/0/180] [13960.074526] LNet: Added LNI 192.168.206.105@tcp112 [8/256/0/180] [13960.792906] LNet: Added LNI 192.168.206.105@tcp113 [8/256/0/180] [13961.661369] LNet: Added LNI 192.168.206.105@tcp114 [8/256/0/180] [13962.560249] LNet: Added LNI 192.168.206.105@tcp115 [8/256/0/180] [13963.395636] LNet: Added LNI 192.168.206.105@tcp116 [8/256/0/180] [13964.139577] LNet: Added LNI 192.168.206.105@tcp117 [8/256/0/180] [13964.935212] LNet: Added LNI 192.168.206.105@tcp118 [8/256/0/180] [13965.767406] LNet: Added LNI 192.168.206.105@tcp119 [8/256/0/180] [13966.544268] LNet: Added LNI 192.168.206.105@tcp120 [8/256/0/180] [13967.320406] LNet: Added LNI 192.168.206.105@tcp121 [8/256/0/180] [13968.031747] LNet: Added LNI 192.168.206.105@tcp122 [8/256/0/180] [13968.903548] LNet: Added LNI 192.168.206.105@tcp123 [8/256/0/180] [13969.753875] LNet: Added LNI 192.168.206.105@tcp124 [8/256/0/180] [13970.480041] LNet: Added LNI 192.168.206.105@tcp125 [8/256/0/180] [13971.225606] LNet: Added LNI 192.168.206.105@tcp126 [8/256/0/180] [13972.297586] LNet: Added LNI 192.168.206.105@tcp127 [8/256/0/180] [13973.508326] LNet: Added LNI 192.168.206.105@tcp128 [8/256/0/180] [13974.555090] LNet: Added LNI 192.168.206.105@tcp129 [8/256/0/180] [13975.664319] LNet: Added LNI 192.168.206.105@tcp130 [8/256/0/180] [13976.739275] LNet: Added LNI 192.168.206.105@tcp131 [8/256/0/180] [13977.705545] LNet: Added LNI 192.168.206.105@tcp132 [8/256/0/180] [13978.582066] LNet: Added LNI 192.168.206.105@tcp133 [8/256/0/180] [13979.501054] LNet: Added LNI 192.168.206.105@tcp134 [8/256/0/180] [13980.305980] LNet: Added LNI 192.168.206.105@tcp135 [8/256/0/180] [13986.983550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13987.017375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13988.129495] Lustre: 463883:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.105@tcp: can't add NID 192.168.206.105@tcp130: rc = -75 [13988.184869] LNetError: 463905:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.105 [13988.234387] LNetError: Protocol error connecting to 192.168.206.105@tcp121 at host 192.168.206.105:988: is it running a compatible version of Lustre? [13988.247900] LNetError: Skipped 2 previous similar messages [13988.395503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13988.432320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13988.550850] Lustre: lustre-MDT0000: new disk, initializing [13988.654943] Lustre: 463941:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.105@tcp: can't add NID 192.168.206.105@tcp130: rc = -75 [13988.674319] Lustre: 463941:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13988.699836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13988.722809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13988.772242] LNet: 1 local NIs in recovery (showing 1): 192.168.206.105@tcp121 [13989.280973] LNetError: 463904:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.105 [13989.293215] LNetError: 463904:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [13989.301077] LNetError: Protocol error connecting to 192.168.206.105@tcp110 at host 192.168.206.105:988: is it running a compatible version of Lustre? [13991.841457] LNetError: 461345:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.105 [13991.849755] LNetError: 461345:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [13991.855568] LNetError: Protocol error connecting to 192.168.206.105@tcp121 at host 192.168.206.105:988: is it running a compatible version of Lustre? [13991.861706] LNetError: Skipped 1 previous similar message [13993.177757] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [13995.943144] LNetError: 463907:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.105 [13995.967371] LNetError: Protocol error connecting to 192.168.206.105@tcp121 at host 192.168.206.105:988: is it running a compatible version of Lustre? [14002.826848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14002.945174] Lustre: 463899:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14003.002729] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14003.015078] Lustre: Skipped 1 previous similar message [14003.105102] LNetError: 461346:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.105 [14003.109798] Lustre: lustre-MDT0001: new disk, initializing [14003.116256] LNetError: Protocol error connecting to 192.168.206.105@tcp121 at host 192.168.206.105:988: is it running a compatible version of Lustre? [14003.147367] Lustre: 464705:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.105@tcp: can't add NID 192.168.206.105@tcp130: rc = -75 [14003.154374] Lustre: 464705:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14003.243501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14003.273860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14003.290100] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14007.331422] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [14017.036431] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14019.430683] LNetError: 463905:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.105 [14019.440987] LNetError: Protocol error connecting to 192.168.206.105@tcp110 at host 192.168.206.105:988: is it running a compatible version of Lustre? [14024.592888] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14032.836657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14033.164938] Lustre: lustre-OST0000: new disk, initializing [14033.173580] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14033.185995] Lustre: 465879:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14033.278453] Lustre: 465884:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.105@tcp: can't add NID 192.168.206.105@tcp130: rc = -75 [14033.293506] Lustre: 465884:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [14033.355859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14036.907744] LNetError: 461345:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.105 [14036.931918] LNetError: 461345:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [14036.964531] LNetError: Protocol error connecting to 192.168.206.105@tcp124 at host 192.168.206.105:988: is it running a compatible version of Lustre? [14036.979964] LNetError: Skipped 5 previous similar messages [14038.568948] Lustre: 466172:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.105@tcp: can't add NID 192.168.206.105@tcp130: rc = -75 [14038.575349] Lustre: 466172:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14038.664597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14038.695073] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14038.835320] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14039.581694] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [14049.183802] LNet: 1 local NIs in recovery (showing 1): 192.168.206.105@tcp121 [14049.294951] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14059.493586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14059.495138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14059.498351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14059.503857] LustreError: Skipped 1 previous similar message [14059.536817] Lustre: Skipped 1 previous similar message [14062.573260] Lustre: server umount lustre-OST0000 complete [14069.730529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14069.731895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14069.748079] Lustre: Skipped 1 previous similar message [14069.752719] Lustre: Skipped 1 previous similar message [14070.689353] LNetError: 461345:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.105 [14070.698721] LNetError: 461345:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [14070.709621] LNetError: Protocol error connecting to 192.168.206.105@tcp110 at host 192.168.206.105:988: is it running a compatible version of Lustre? [14070.722518] LNetError: Skipped 10 previous similar messages [14072.161029] Lustre: server umount lustre-MDT0000 complete [14075.320293] LustreError: 463892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855409 with bad export cookie 554978251552198756 [14075.324943] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14075.331953] LustreError: 463892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14075.632431] Lustre: server umount lustre-MDT0001 complete [14085.285905] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [14088.354076] Key type lgssc unregistered [14088.807256] LNet: 467794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14088.814642] LNetError: 467794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14088.833557] LNet: Removed LNI 192.168.206.105@tcp [14089.887530] LNet: Removed LNI 192.168.206.105@tcp100 [14089.895544] LNetError: 467794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14089.912486] LNetError: 467794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14089.924035] LNet: Removed LNI 192.168.206.105@tcp101 [14090.975544] LNet: Removed LNI 192.168.206.105@tcp102 [14090.989635] LNetError: 467794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14091.007496] LNetError: 467794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14091.025959] LNet: Removed LNI 192.168.206.105@tcp103 [14092.063141] LNet: Removed LNI 192.168.206.105@tcp104 [14093.087833] LNet: Removed LNI 192.168.206.105@tcp105 [14093.095854] LNetError: 467794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14093.111985] LNetError: 467794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14093.140500] LNet: Removed LNI 192.168.206.105@tcp106 [14094.175180] LNet: Removed LNI 192.168.206.105@tcp107 [14095.199184] LNet: Removed LNI 192.168.206.105@tcp108 [14096.223629] LNet: Removed LNI 192.168.206.105@tcp109 [14097.247151] LNet: Removed LNI 192.168.206.105@tcp110 [14097.251681] LNetError: 467794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14097.264573] LNetError: 467794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14097.272618] LNet: Removed LNI 192.168.206.105@tcp111 [14098.335811] LNet: Removed LNI 192.168.206.105@tcp112 [14099.370034] LNet: Removed LNI 192.168.206.105@tcp113 [14100.383087] LNet: Removed LNI 192.168.206.105@tcp114 [14101.407152] LNet: Removed LNI 192.168.206.105@tcp115 [14102.431163] LNet: Removed LNI 192.168.206.105@tcp116 [14103.455303] LNet: Removed LNI 192.168.206.105@tcp117 [14104.480508] LNet: Removed LNI 192.168.206.105@tcp118 [14105.503365] LNet: Removed LNI 192.168.206.105@tcp119 [14105.513661] LNetError: 467794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14105.540115] LNetError: 467794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14105.568826] LNet: Removed LNI 192.168.206.105@tcp120 [14105.579896] LNet: Removed LNI 192.168.206.105@tcp121 [14106.591145] LNet: Removed LNI 192.168.206.105@tcp122 [14107.615360] LNet: Removed LNI 192.168.206.105@tcp123 [14108.639185] LNet: Removed LNI 192.168.206.105@tcp124 [14109.666528] LNet: Removed LNI 192.168.206.105@tcp125 [14110.687290] LNet: Removed LNI 192.168.206.105@tcp126 [14111.711312] LNet: Removed LNI 192.168.206.105@tcp127 [14112.735231] LNet: Removed LNI 192.168.206.105@tcp128 [14113.759161] LNet: Removed LNI 192.168.206.105@tcp129 [14113.763704] LNet: Removed LNI 192.168.206.105@tcp130 [14113.768141] LNet: Removed LNI 192.168.206.105@tcp131 [14113.772413] LNet: Removed LNI 192.168.206.105@tcp132 [14113.779825] LNet: Removed LNI 192.168.206.105@tcp133 [14113.790939] LNet: Removed LNI 192.168.206.105@tcp134 [14113.813324] LNet: Removed LNI 192.168.206.105@tcp135 [14115.148859] Key type .llcrypt unregistered [14115.150943] Key type ._llcrypt unregistered [14137.165748] Key type ._llcrypt registered [14137.170411] Key type .llcrypt registered [14137.312645] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [14151.035556] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [14152.475372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14152.491741] alg: No test for adler32 (adler32-zlib) [14153.531349] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [14153.704380] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [14155.383681] Key type lgssc registered [14156.537091] Lustre: Echo OBD driver; http://www.lustre.org/ [14193.108508] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 14:32:05 (1787855525) [14211.070992] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [14214.118062] Key type lgssc unregistered [14214.457612] LNet: 471884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14214.463741] LNetError: 471884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14214.485432] LNet: Removed LNI 192.168.206.105@tcp [14215.203765] Key type .llcrypt unregistered [14215.208572] Key type ._llcrypt unregistered [14232.595129] Key type ._llcrypt registered [14232.598845] Key type .llcrypt registered [14232.721589] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [14234.802907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14234.868437] alg: No test for adler32 (adler32-zlib) [14236.160730] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [14236.482791] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [14238.199168] Key type lgssc registered [14239.526272] Lustre: Echo OBD driver; http://www.lustre.org/ [14249.382153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14249.398597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14250.719953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14250.752946] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14250.878288] Lustre: lustre-MDT0000: new disk, initializing [14251.002579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14251.024052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14255.683528] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [14266.375513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14266.552630] Lustre: 473100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14266.623551] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14266.627598] Lustre: Skipped 1 previous similar message [14266.764244] Lustre: lustre-MDT0001: new disk, initializing [14266.929075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14267.007517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14267.017180] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14271.777949] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [14281.309878] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14286.604965] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14293.832969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14294.146163] Lustre: lustre-OST0000: new disk, initializing [14294.153644] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14294.159786] Lustre: 475070:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14294.293590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14299.760382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14299.789121] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14299.886607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14301.438624] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [14312.202608] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14317.298411] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing lnet_if_list [14323.490828] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14323.524041] LNet: Added LNI 192.168.206.105@tcp10 [8/256/0/180] [14329.853993] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14329.880531] LNet: Added LNI 192.168.206.105@tcp11 [8/256/0/180] [14334.618352] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14334.667612] LNet: Added LNI 192.168.206.105@tcp12 [8/256/0/180] [14339.078369] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14339.119924] LNet: Added LNI 192.168.206.105@tcp13 [8/256/0/180] [14344.109088] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14344.139268] LNet: Added LNI 192.168.206.105@tcp14 [8/256/0/180] [14349.381632] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14349.426615] LNet: Added LNI 192.168.206.105@tcp15 [8/256/0/180] [14354.359271] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14354.381939] LNet: Added LNI 192.168.206.105@tcp16 [8/256/0/180] [14359.677349] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14359.707920] LNet: Added LNI 192.168.206.105@tcp17 [8/256/0/180] [14364.825844] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14364.854787] LNet: Added LNI 192.168.206.105@tcp18 [8/256/0/180] [14370.044421] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14370.091929] LNet: Added LNI 192.168.206.105@tcp19 [8/256/0/180] [14375.895210] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14375.917409] LNet: Added LNI 192.168.206.105@tcp20 [8/256/0/180] [14382.532298] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14382.572741] LNet: Added LNI 192.168.206.105@tcp21 [8/256/0/180] [14390.673986] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14390.726833] LNet: Added LNI 192.168.206.105@tcp22 [8/256/0/180] [14398.162400] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14398.201474] LNet: Added LNI 192.168.206.105@tcp23 [8/256/0/180] [14405.507685] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14405.538085] LNet: Added LNI 192.168.206.105@tcp24 [8/256/0/180] [14412.788157] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14412.870046] LNet: Added LNI 192.168.206.105@tcp25 [8/256/0/180] [14419.936340] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14419.959224] LNet: Added LNI 192.168.206.105@tcp26 [8/256/0/180] [14426.879838] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14426.911845] LNet: Added LNI 192.168.206.105@tcp27 [8/256/0/180] [14433.466887] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14433.496082] LNet: Added LNI 192.168.206.105@tcp28 [8/256/0/180] [14440.627107] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14440.667995] LNet: Added LNI 192.168.206.105@tcp29 [8/256/0/180] [14447.609793] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14447.640822] LNet: Added LNI 192.168.206.105@tcp30 [8/256/0/180] [14453.567601] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14453.601179] LNet: Added LNI 192.168.206.105@tcp31 [8/256/0/180] [14459.398949] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14459.435896] LNet: Added LNI 192.168.206.105@tcp32 [8/256/0/180] [14466.316934] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14466.349762] LNet: Added LNI 192.168.206.105@tcp33 [8/256/0/180] [14473.800954] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14473.843528] LNet: Added LNI 192.168.206.105@tcp34 [8/256/0/180] [14481.465319] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14481.500276] LNet: Added LNI 192.168.206.105@tcp35 [8/256/0/180] [14488.280378] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14488.309869] LNet: Added LNI 192.168.206.105@tcp36 [8/256/0/180] [14496.675939] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14496.706945] LNet: Added LNI 192.168.206.105@tcp37 [8/256/0/180] [14505.875713] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14505.911775] LNet: Added LNI 192.168.206.105@tcp38 [8/256/0/180] [14515.099728] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14515.144446] LNet: Added LNI 192.168.206.105@tcp39 [8/256/0/180] [14523.675375] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14523.722981] LNet: Added LNI 192.168.206.105@tcp40 [8/256/0/180] [14531.989057] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14532.031568] LNet: Added LNI 192.168.206.105@tcp41 [8/256/0/180] [14540.945486] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14541.006542] LNet: Added LNI 192.168.206.105@tcp42 [8/256/0/180] [14548.416362] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14548.436940] LNet: Added LNI 192.168.206.105@tcp43 [8/256/0/180] [14555.833442] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14555.892333] LNet: Added LNI 192.168.206.105@tcp44 [8/256/0/180] [14564.326659] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14564.353535] LNet: Added LNI 192.168.206.105@tcp45 [8/256/0/180] [14572.172078] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14572.207690] LNet: Added LNI 192.168.206.105@tcp46 [8/256/0/180] [14579.457390] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14579.480147] LNet: Added LNI 192.168.206.105@tcp47 [8/256/0/180] [14587.644132] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14587.703864] LNet: Added LNI 192.168.206.105@tcp48 [8/256/0/180] [14595.382896] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14595.406966] LNet: Added LNI 192.168.206.105@tcp49 [8/256/0/180] [14604.042943] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14604.067046] LNet: Added LNI 192.168.206.105@tcp50 [8/256/0/180] [14622.690732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14622.691638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14622.699554] Lustre: Skipped 1 previous similar message [14622.710494] Lustre: Skipped 2 previous similar messages [14627.810588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14627.820171] Lustre: Skipped 2 previous similar messages [14628.184843] Lustre: server umount lustre-MDT0000 complete [14631.407789] LustreError: 473092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855965 with bad export cookie 8093516252875873924 [14631.417876] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14631.687331] Lustre: server umount lustre-MDT0001 complete [14648.287432] Lustre: 472372:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787855966/real 1787855966] req@000000000f4951c0 x1874702442341248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787855982 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14648.353331] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14648.378519] Lustre: Skipped 1 previous similar message [14653.147482] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [14653.151376] Lustre: 472371:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787855970/real 1787855970] req@0000000021fbbf81 x1874702442341376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787855986 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14657.504231] Lustre: 472369:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787855975/real 1787855975] req@000000001176cb64 x1874702442341760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787855991 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14657.533678] Lustre: 472369:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14661.974942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14663.647150] Lustre: 472372:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787855981/real 1787855981] req@00000000e99eaf94 x1874702442341888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787855997 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14677.088543] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7051f226948468a9 [14677.105369] Lustre: MGC192.168.206.105@tcp: Connection restored to 0@lo (at 0@lo) [14677.458736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14678.500611] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14682.115360] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [14684.642224] LustreError: 475062:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14684.666533] LustreError: 475062:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [14689.763378] LustreError: 475061:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14691.097153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14691.397791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14693.418456] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14696.232673] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [14704.608446] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14711.249738] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14733.792355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14733.801776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14733.823103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14739.431346] LustreError: 475052:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14739.441644] LustreError: 475052:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [14739.601462] Lustre: server umount lustre-OST0000 complete [14744.545617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14744.554105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14744.560385] Lustre: Skipped 2 previous similar messages [14744.586416] Lustre: Skipped 3 previous similar messages [14749.665484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14749.680067] Lustre: Skipped 1 previous similar message [14750.119707] Lustre: server umount lustre-MDT0000 complete [14754.787756] LustreError: 499321:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14754.809778] LustreError: 499321:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [14755.184426] LustreError: 477754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856089 with bad export cookie 8093516252875876521 [14755.198786] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14755.208084] LustreError: 477754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [14755.740804] Lustre: server umount lustre-MDT0001 complete [14769.826519] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [14772.998267] Key type lgssc unregistered [14773.607233] LNet: 502120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14773.636043] LNetError: 502120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14773.648878] LNet: Removed LNI 192.168.206.105@tcp [14773.652925] LNet: Removed LNI 192.168.206.105@tcp10 [14773.663365] LNet: Removed LNI 192.168.206.105@tcp11 [14773.669883] LNet: Removed LNI 192.168.206.105@tcp12 [14773.674108] LNet: Removed LNI 192.168.206.105@tcp13 [14773.679754] LNet: Removed LNI 192.168.206.105@tcp14 [14773.685760] LNet: Removed LNI 192.168.206.105@tcp15 [14773.693148] LNet: Removed LNI 192.168.206.105@tcp16 [14773.695747] LNet: Removed LNI 192.168.206.105@tcp17 [14773.698917] LNet: Removed LNI 192.168.206.105@tcp18 [14773.701933] LNet: Removed LNI 192.168.206.105@tcp19 [14773.706754] LNet: Removed LNI 192.168.206.105@tcp20 [14773.708756] LNet: Removed LNI 192.168.206.105@tcp21 [14773.713355] LNet: Removed LNI 192.168.206.105@tcp22 [14773.719655] LNet: Removed LNI 192.168.206.105@tcp23 [14773.724832] LNet: Removed LNI 192.168.206.105@tcp24 [14773.727755] LNet: Removed LNI 192.168.206.105@tcp25 [14773.730836] LNet: Removed LNI 192.168.206.105@tcp26 [14773.734486] LNet: Removed LNI 192.168.206.105@tcp27 [14773.737226] LNet: Removed LNI 192.168.206.105@tcp28 [14773.739809] LNet: Removed LNI 192.168.206.105@tcp29 [14773.742651] LNet: Removed LNI 192.168.206.105@tcp30 [14773.744746] LNet: Removed LNI 192.168.206.105@tcp31 [14773.746995] LNet: Removed LNI 192.168.206.105@tcp32 [14773.750133] LNet: Removed LNI 192.168.206.105@tcp33 [14773.753240] LNet: Removed LNI 192.168.206.105@tcp34 [14773.755280] LNet: Removed LNI 192.168.206.105@tcp35 [14773.758256] LNet: Removed LNI 192.168.206.105@tcp36 [14773.760748] LNet: Removed LNI 192.168.206.105@tcp37 [14773.762427] LNet: Removed LNI 192.168.206.105@tcp38 [14773.764467] LNet: Removed LNI 192.168.206.105@tcp39 [14773.767176] LNet: Removed LNI 192.168.206.105@tcp40 [14773.769436] LNet: Removed LNI 192.168.206.105@tcp41 [14773.776443] LNet: Removed LNI 192.168.206.105@tcp42 [14773.781755] LNet: Removed LNI 192.168.206.105@tcp43 [14773.785741] LNet: Removed LNI 192.168.206.105@tcp44 [14773.788201] LNet: Removed LNI 192.168.206.105@tcp45 [14773.791865] LNet: Removed LNI 192.168.206.105@tcp46 [14773.795203] LNet: Removed LNI 192.168.206.105@tcp47 [14773.797490] LNet: Removed LNI 192.168.206.105@tcp48 [14773.801746] LNet: Removed LNI 192.168.206.105@tcp49 [14773.812869] LNet: Removed LNI 192.168.206.105@tcp50 [14774.741667] Key type .llcrypt unregistered [14774.748320] Key type ._llcrypt unregistered [14798.791936] Key type ._llcrypt registered [14798.798352] Key type .llcrypt registered [14798.930764] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [14817.798636] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [14819.113803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14819.128873] alg: No test for adler32 (adler32-zlib) [14820.181488] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [14820.459779] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [14822.167317] Key type lgssc registered [14823.746138] Lustre: Echo OBD driver; http://www.lustre.org/ [14865.819283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14865.888174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14867.217126] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14867.250355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14867.345571] Lustre: lustre-MDT0000: new disk, initializing [14867.436124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14867.464881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14872.770033] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [14885.285960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14885.446987] Lustre: 505321:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14885.565245] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14885.577508] Lustre: Skipped 1 previous similar message [14885.680188] Lustre: lustre-MDT0001: new disk, initializing [14885.792630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14885.853343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14885.865961] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14890.871569] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [14901.758840] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14908.992831] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14917.176580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14917.443531] Lustre: *** cfs_fail_loc=90f, val=0*** [14917.576154] Lustre: lustre-OST0000: new disk, initializing [14917.579311] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14917.584462] Lustre: 507340:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14917.701530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14924.307415] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [14925.464135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14925.475382] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14925.601781] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14934.556084] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14946.279620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14946.299702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14946.303241] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14947.563680] Lustre: server umount lustre-OST0000 complete [14952.416712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14952.429367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14952.441645] Lustre: Skipped 1 previous similar message [14952.448026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14952.459135] Lustre: Skipped 1 previous similar message [14956.513046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14956.515344] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14957.895910] Lustre: server umount lustre-MDT0000 complete [14961.774194] LustreError: 506118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856295 with bad export cookie 722721902069700613 [14961.789042] LustreError: 506118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14961.792925] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14962.273201] Lustre: server umount lustre-MDT0001 complete [14975.080782] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [14978.391903] Key type lgssc unregistered [14978.723802] LNet: 509249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14978.733366] LNetError: 509249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14978.763713] LNet: Removed LNI 192.168.206.105@tcp [14979.639786] Key type .llcrypt unregistered [14979.644767] Key type ._llcrypt unregistered [14999.957938] Key type ._llcrypt registered [14999.961608] Key type .llcrypt registered [15000.088639] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [15000.878542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15000.967575] alg: No test for adler32 (adler32-zlib) [15002.135729] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [15002.385709] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [15004.127214] Key type lgssc registered [15005.591398] Lustre: Echo OBD driver; http://www.lustre.org/ [15014.590634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15014.617454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15016.112147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15021.980893] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [15025.635451] LustreError: 510520:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15025.648889] LustreError: 510520:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [15030.753514] LustreError: 510519:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15032.082756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15032.454154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15037.130160] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [15045.414360] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15051.867993] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15059.598498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15059.939745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15067.444569] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [15077.522855] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15090.486750] loop0: detected capacity change from 0 to 204800000 [15090.697411] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15090.710990] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15090.724495] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15097.057865] loop0: detected capacity change from 0 to 204800000 [15097.326282] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15097.338075] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15097.350851] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15102.519550] loop0: detected capacity change from 0 to 204800000 [15107.883164] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15108.006095] Lustre: fs15246-OST0000: new disk, initializing [15108.011692] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15108.020569] Lustre: 514114:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15108.121800] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15114.585150] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [15124.702834] loop1: detected capacity change from 0 to 204800000 [15131.270025] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15131.438323] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15131.509439] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15131.660728] Lustre: fs15246-MDT0000: new disk, initializing [15131.883602] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15131.916977] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15132.023960] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15133.088923] Lustre: 515236:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000001c2376c0 x1874703245428224/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787856482 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15137.051485] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [15142.254921] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15142.263658] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15142.278721] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15161.318114] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15161.337428] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15165.717371] Lustre: server umount fs15246-MDT0000 complete [15166.446960] LustreError: 511760:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15171.566347] LustreError: 512636:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15176.679285] LustreError: 512636:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15177.485906] Lustre: server umount fs15246-OST0000 complete [15192.037664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15192.064388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15192.069044] Lustre: Skipped 1 previous similar message [15197.157283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15197.176245] Lustre: Skipped 1 previous similar message [15197.721644] Lustre: server umount lustre-OST0000 complete [15202.275166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15202.301535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15202.301759] Lustre: Skipped 2 previous similar messages [15202.322966] Lustre: Skipped 1 previous similar message [15207.396956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15207.410466] Lustre: Skipped 1 previous similar message [15208.608552] Lustre: server umount lustre-MDT0000 complete [15212.513616] LustreError: 516050:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15212.543778] LustreError: 516050:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [15212.969715] LustreError: 510501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856546 with bad export cookie 15416699791901859188 [15212.978294] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15212.980302] LustreError: 510501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15213.488251] Lustre: server umount lustre-MDT0001 complete [15224.685556] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [15227.678539] Key type lgssc unregistered [15227.996805] LNet: 518337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15228.007426] LNetError: 518337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15228.026291] LNet: Removed LNI 192.168.206.105@tcp [15228.953045] Key type .llcrypt unregistered [15228.955053] Key type ._llcrypt unregistered [15248.711254] Key type ._llcrypt registered [15248.713899] Key type .llcrypt registered [15248.834501] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [15249.903728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15250.075595] alg: No test for adler32 (adler32-zlib) [15251.207151] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [15251.416592] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [15253.055170] Key type lgssc registered [15254.040991] Lustre: Echo OBD driver; http://www.lustre.org/ [15290.270390] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [15301.028843] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [15338.900214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15338.922470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15340.087188] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15340.115483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15340.201226] Lustre: lustre-MDT0000: new disk, initializing [15340.291536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15340.308927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15344.442658] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [15355.239981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15355.399816] Lustre: 522804:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15355.478340] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15355.482043] Lustre: Skipped 1 previous similar message [15355.613529] Lustre: lustre-MDT0001: new disk, initializing [15355.705238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15355.743161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15355.753616] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15360.898919] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [15371.800693] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15378.109778] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15385.341668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15385.687393] Lustre: lustre-OST0000: new disk, initializing [15385.693603] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15385.704905] Lustre: 524775:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15385.854339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15391.957691] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [15394.359042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15394.371122] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15394.467170] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15401.348173] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15405.820435] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15406.006365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15410.420847] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15410.632100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15414.753530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15414.758847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15414.772624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15415.264316] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15415.272529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15415.280757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15415.292831] Lustre: Skipped 1 previous similar message [15418.658549] Lustre: server umount lustre-OST0000 complete [15422.435366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15422.450659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15422.463740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15425.506866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15425.508257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15428.305576] Lustre: server umount lustre-MDT0000 complete [15431.881939] LustreError: 522798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856765 with bad export cookie 16768530212691538234 [15431.885587] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15431.893542] LustreError: 522798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15432.106676] Lustre: server umount lustre-MDT0001 complete [15457.784653] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [15467.917324] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [15496.238540] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15225 sec ========= 14:53:48 (1787856828) [15497.644924] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:53:49 (1787856829) === [15500.361768] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:53:52 (1787856832) === [15541.115438] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [15543.990494] Key type lgssc unregistered [15544.276513] LNet: 531421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15544.294155] LNetError: 531421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15545.317205] LNet: Removed LNI 192.168.206.105@tcp [15546.237835] Key type .llcrypt unregistered [15546.242746] Key type ._llcrypt unregistered