[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 497884785 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002383] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.006998] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.010123] LSM: Security Framework initializing [ 0.011043] Yama: becoming mindful. [ 0.012025] SELinux: Initializing. [ 0.013053] *** VALIDATE selinux *** [ 0.020811] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024713] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026138] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027125] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028139] *** VALIDATE tmpfs *** [ 0.029541] *** VALIDATE proc *** [ 0.031132] *** VALIDATE cgroup *** [ 0.032012] *** VALIDATE cgroup2 *** [ 0.034038] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035168] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037027] Spectre V2 : User space: Vulnerable [ 0.038021] Speculative Store Bypass: Vulnerable [ 0.041337] debug: unmapping init [mem 0xffffffffb3459000-0xffffffffb3460fff] [ 0.044000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044665] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045038] ... version: 2 [ 0.046014] ... bit width: 48 [ 0.047011] ... generic registers: 4 [ 0.048011] ... value mask: 0000ffffffffffff [ 0.049014] ... max period: 00007fffffffffff [ 0.050011] ... fixed-purpose events: 3 [ 0.051011] ... event mask: 000000070000000f [ 0.052278] rcu: Hierarchical SRCU implementation. [ 0.054810] smp: Bringing up secondary CPUs ... [ 0.055584] x86: Booting SMP configuration: [ 0.056024] .... node #0, CPUs: #1 #2 #3 [ 0.059080] smp: Brought up 1 node, 4 CPUs [ 0.061014] smpboot: Max logical packages: 1 [ 0.062020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.149439] node 0 deferred pages initialised in 85ms [ 0.152142] devtmpfs: initialized [ 0.153237] x86/mm: Memory block size: 128MB [ 0.155782] gcov: version magic: 0x41383552 [ 0.158035] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160325] pinctrl core: initialized pinctrl subsystem [ 0.161253] [ 0.161816] ************************************************************* [ 0.162014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163022] ** ** [ 0.164021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165014] ** ** [ 0.166020] ** This means that this kernel is built to expose internal ** [ 0.167019] ** IOMMU data structures, which may compromise security on ** [ 0.168013] ** your system. ** [ 0.169008] ** ** [ 0.170008] ** If you see this message and you are not debugging the ** [ 0.171012] ** kernel, report this immediately to your vendor! ** [ 0.172008] ** ** [ 0.173011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174009] ************************************************************* [ 0.175554] NET: Registered protocol family 16 [ 0.176328] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.177076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.178068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.179483] cpuidle: using governor menu [ 0.181588] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.184544] PCI: Using configuration type 1 for base access [ 0.186134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.194187] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.197090] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.201042] cryptd: max_cpu_qlen set to 1000 [ 0.203225] ACPI: Added _OSI(Module Device) [ 0.205014] ACPI: Added _OSI(Processor Device) [ 0.207014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.208013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.211000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.218550] ACPI: Interpreter enabled [ 0.220057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.221017] ACPI: Using IOAPIC for interrupt routing [ 0.223107] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.226382] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.236291] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.239040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.241018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.244096] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.249631] acpiphp: Slot [2] registered [ 0.250152] acpiphp: Slot [5] registered [ 0.252088] acpiphp: Slot [6] registered [ 0.253109] acpiphp: Slot [7] registered [ 0.253985] acpiphp: Slot [8] registered [ 0.255107] acpiphp: Slot [9] registered [ 0.256132] acpiphp: Slot [10] registered [ 0.257142] acpiphp: Slot [3] registered [ 0.259079] acpiphp: Slot [4] registered [ 0.260109] acpiphp: Slot [11] registered [ 0.261104] acpiphp: Slot [12] registered [ 0.262083] acpiphp: Slot [13] registered [ 0.264101] acpiphp: Slot [14] registered [ 0.265128] acpiphp: Slot [15] registered [ 0.267102] acpiphp: Slot [16] registered [ 0.268091] acpiphp: Slot [17] registered [ 0.269137] acpiphp: Slot [18] registered [ 0.271091] acpiphp: Slot [19] registered [ 0.272104] acpiphp: Slot [20] registered [ 0.274096] acpiphp: Slot [21] registered [ 0.275135] acpiphp: Slot [22] registered [ 0.276099] acpiphp: Slot [23] registered [ 0.278103] acpiphp: Slot [24] registered [ 0.279135] acpiphp: Slot [25] registered [ 0.281110] acpiphp: Slot [26] registered [ 0.282084] acpiphp: Slot [27] registered [ 0.283115] acpiphp: Slot [28] registered [ 0.284087] acpiphp: Slot [29] registered [ 0.285084] acpiphp: Slot [30] registered [ 0.287119] acpiphp: Slot [31] registered [ 0.288095] PCI host bridge to bus 0000:00 [ 0.289021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.291023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.294023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.296023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.298025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.301040] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.303191] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.306438] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.309402] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.319026] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.324506] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.327023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.329018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.331019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.333325] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.336696] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.338023] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.339496] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.344014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.357015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.361014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.365651] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.372018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.378020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.402018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.411950] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.426031] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.435032] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.449027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.458463] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.466022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.473023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.492020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.503253] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.512034] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.524032] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.542028] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.555042] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.565021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.578022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.605029] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.620265] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.630033] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.640039] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.657032] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.671822] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.673400] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.677405] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.680399] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.682256] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.687178] iommu: Default domain type: Passthrough [ 0.689434] SCSI subsystem initialized [ 0.690124] ACPI: bus type USB registered [ 0.692142] usbcore: registered new interface driver usbfs [ 0.694110] usbcore: registered new interface driver hub [ 0.696090] usbcore: registered new device driver usb [ 0.698223] pps_core: LinuxPPS API ver. 1 registered [ 0.700014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.703077] PTP clock support registered [ 0.705099] EDAC MC: Ver: 3.0.0 [ 0.707180] PCI: Using ACPI for IRQ routing [ 0.709953] NetLabel: Initializing [ 0.711013] NetLabel: domain hash size = 128 [ 0.712009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.714089] NetLabel: unlabeled traffic allowed by default [ 0.716158] vgaarb: loaded [ 0.718078] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.719011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.722302] clocksource: Switched to clocksource kvm-clock [ 0.831205] VFS: Disk quotas dquot_6.6.0 [ 0.832774] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.834437] *** VALIDATE ramfs *** [ 0.835747] *** VALIDATE hugetlbfs *** [ 0.837202] pnp: PnP ACPI init [ 0.839530] pnp: PnP ACPI: found 6 devices [ 0.858056] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.860849] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.862772] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.864609] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.866882] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.868712] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.870644] NET: Registered protocol family 2 [ 0.872880] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.877821] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.882290] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.888247] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.892215] TCP: Hash tables configured (established 65536 bind 65536) [ 0.895385] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.898486] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.900687] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.903810] NET: Registered protocol family 1 [ 0.906720] RPC: Registered named UNIX socket transport module. [ 0.908938] RPC: Registered udp transport module. [ 0.910666] RPC: Registered tcp transport module. [ 0.912218] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.914538] NET: Registered protocol family 44 [ 0.916133] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.918331] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.920441] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.922737] PCI: CLS 0 bytes, default 64 [ 0.925189] Unpacking initramfs... [ 2.334951] debug: unmapping init [mem 0xffff9bdbbcc54000-0xffff9bdbbffbffff] [ 2.338883] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.341251] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.344148] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.867611] Initialise system trusted keyrings [ 2.869346] Key type blacklist registered [ 2.871270] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.896255] zbud: loaded [ 2.904822] *** VALIDATE nfs *** [ 2.906569] *** VALIDATE nfs4 *** [ 2.908516] pstore: using deflate compression [ 2.914971] Platform Keyring initialized [ 3.066787] NET: Registered protocol family 38 [ 3.069979] Key type asymmetric registered [ 3.071087] Asymmetric key parser 'x509' registered [ 3.072754] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.076085] io scheduler mq-deadline registered [ 3.077908] io scheduler kyber registered [ 3.079894] io scheduler bfq registered [ 3.081821] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.085145] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.088238] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.090261] ACPI: Power Button [PWRF] [ 3.095219] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.101715] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.113896] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.122939] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.147586] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.195122] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.230554] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.236183] Non-volatile memory driver v1.3 [ 3.238102] Linux agpgart interface v0.103 [ 3.269695] virtio_blk virtio1: [vda] 146680 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.272601] vda: detected capacity change from 0 to 75100160 [ 3.289829] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.293097] vdb: detected capacity change from 0 to 1073741824 [ 3.309018] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.312063] vdc: detected capacity change from 0 to 2621440000 [ 3.326150] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.328857] vdd: detected capacity change from 0 to 2621440000 [ 3.341765] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.344323] vde: detected capacity change from 0 to 4294967296 [ 3.357702] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.360661] vdf: detected capacity change from 0 to 4294967296 [ 3.366938] libphy: Fixed MDIO Bus: probed [ 3.372724] usbcore: registered new interface driver usbserial_generic [ 3.375166] usbserial: USB Serial support registered for generic [ 3.377100] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.381738] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.383548] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.386228] mousedev: PS/2 mouse device common for all mice [ 3.388875] rtc_cmos 00:05: RTC can wake from S4 [ 3.392947] rtc_cmos 00:05: registered as rtc0 [ 3.394815] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.398608] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.401640] intel_pstate: CPU model not supported [ 3.404626] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.408919] hid: raw HID events driver (C) Jiri Kosina [ 3.411151] usbcore: registered new interface driver usbhid [ 3.413286] usbhid: USB HID core driver [ 3.414992] drop_monitor: Initializing network drop monitor service [ 3.417345] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.417659] Initializing XFRM netlink socket [ 3.422884] NET: Registered protocol family 10 [ 3.426710] Segment Routing with IPv6 [ 3.428216] NET: Registered protocol family 17 [ 3.430452] mpls_gso: MPLS GSO support [ 3.435416] RAS: Correctable Errors collector initialized. [ 3.437653] AVX version of gcm_enc/dec engaged. [ 3.439392] AES CTR mode by8 optimization enabled [ 3.525692] sched_clock: Marking stable (3525665055, 0)->(4503680798, -978015743) [ 3.529532] registered taskstats version 1 [ 3.532392] Loading compiled-in X.509 certificates [ 3.534726] zswap: loaded using pool lzo/zbud [ 3.564237] Key type big_key registered [ 3.577824] Key type encrypted registered [ 3.579622] ima: No TPM chip found, activating TPM-bypass! [ 3.581621] ima: Allocated hash algorithm: sha1 [ 3.583337] ima: No architecture policies found [ 3.585308] evm: Initialising EVM extended attributes: [ 3.586886] evm: security.selinux [ 3.588058] evm: security.ima [ 3.588873] evm: security.capability [ 3.589951] evm: HMAC attrs: 0x1 [ 3.592371] rtc_cmos 00:05: setting system clock to 2026-08-27 09:56:18 UTC (1787824578) [ 3.599533] debug: unmapping init [mem 0xffffffffb4403000-0xffffffffb45fffff] [ 3.602679] debug: unmapping init [mem 0xffffffffb3182000-0xffffffffb3458fff] [ 3.612138] Write protecting the kernel read-only data: 28672k [ 3.615878] debug: unmapping init [mem 0xffffffffb1803000-0xffffffffb19fffff] [ 3.618405] debug: unmapping init [mem 0xffffffffb2114000-0xffffffffb21fffff] [ 3.654640] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.664448] systemd[1]: Detected virtualization kvm. [ 3.665842] systemd[1]: Detected architecture x86-64. [ 3.667404] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.693373] systemd[1]: No hostname configured. [ 3.694888] systemd[1]: Set hostname to . [ 3.696990] random: systemd: uninitialized urandom read (16 bytes read) [ 3.699230] systemd[1]: Initializing machine ID from random generator. [ 3.758632] random: ln: uninitialized urandom read (6 bytes read) [ 3.840462] random: systemd: uninitialized urandom read (16 bytes read) [ 3.843551] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.853341] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ 3.862608] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.376839] random: fast init done [ 4.437063] device-mapper: uevent: version 1.0.3 [ 4.439547] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.291034] virtio_net virtio0 ens2: renamed from eth0 [ 5.400575] scsi host0: ata_piix [ 5.428931] scsi host1: ata_piix [ 5.430102] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.431805] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.175416] dracut-initqueue[584]: RTNETLINK answers: File exists [ 10.094910] random: crng init done [ 10.096283] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.597267] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.769204] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.987769] SELinux: Disabled at runtime. [ 12.046777] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.052221] systemd[1]: Detected virtualization kvm. [ 12.053495] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.587469] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.591379] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.596389] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.599288] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.602474] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.608035] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.613782] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ 12.642385] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.110218] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.443440] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.529670] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.671693] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.685077] EDAC sbridge: Ver: 1.1.2 [ 15.554669] Key type dns_resolver registered [ 15.884509] NFS: Registering the id_resolver key type [ 15.886167] Key type id_resolver registered [ 15.887612] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg106-server login: [ 78.576168] libcfs: loading out-of-tree module taints kernel. [ 78.631182] Key type ._llcrypt registered [ 78.633529] Key type .llcrypt registered [ 78.736727] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 100.383467] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 102.083177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 102.123539] alg: No test for adler32 (adler32-zlib) [ 103.638973] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 105.034565] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 106.855365] Key type lgssc registered [ 109.087327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 123.140702] hrtimer: interrupt took 2799310 ns [ 127.358192] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 170.246585] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 184.090082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 184.205033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 185.854549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 185.894986] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 186.096762] Lustre: lustre-MDT0000: new disk, initializing [ 186.226902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.286148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 191.682916] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.569469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 207.751296] Lustre: 6513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 207.779979] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 207.783700] Lustre: Skipped 1 previous similar message [ 207.892961] Lustre: lustre-MDT0001: new disk, initializing [ 208.024497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 208.063518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 208.081299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 213.411706] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.497627] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 228.621377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 228.843949] Lustre: lustre-OST0000: new disk, initializing [ 228.851381] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 228.858107] Lustre: 8451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 228.933162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 229.063713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 229.087065] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 229.170402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 235.622577] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 255.503180] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 255.636536] Lustre: lustre-OST0001: new disk, initializing [ 255.639790] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 255.644845] Lustre: 9525:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 255.704239] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 262.191597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 262.204139] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 262.342296] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 262.800514] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 275.751156] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 282.298223] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 289.638669] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing check_logdir /tmp/testlogs/ [ 295.001894] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing yml_node [ 300.000886] Lustre: DEBUG MARKER: Client: 2.17.57.69 [ 302.808111] Lustre: DEBUG MARKER: MDS: 2.17.57.69 [ 305.640429] Lustre: DEBUG MARKER: OSS: 2.17.57.69 [ 307.840785] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 06:01:20 EDT 2026 [ 326.682789] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 328.688517] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 346.591912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 346.599321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.616806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 349.664360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 349.667219] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 354.848515] Lustre: server umount lustre-MDT0000 complete [ 359.686607] LustreError: 7927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824934 with bad export cookie 8263992823009849014 [ 359.695974] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 359.711740] LustreError: 7927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 359.903681] LustreError: 8941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 359.903757] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.925288] LustreError: 8941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 359.953447] Lustre: Skipped 2 previous similar messages [ 359.956343] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 359.978932] Lustre: Skipped 4 previous similar messages [ 365.025813] LustreError: 8941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 365.029115] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 365.061951] Lustre: Skipped 1 previous similar message [ 365.064550] LustreError: 8941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 366.649565] Lustre: server umount lustre-MDT0001 complete [ 371.247259] Lustre: server umount lustre-OST0000 complete [ 375.833499] Lustre: server umount lustre-OST0001 complete [ 385.609793] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 396.855995] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 444.443167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 444.646313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 444.667680] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 444.722404] Lustre: lustre-MDT0000: new disk, initializing [ 444.772623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 444.801308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 449.432251] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 459.989942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 460.137234] Lustre: 15096:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 460.212222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 460.218568] Lustre: Skipped 1 previous similar message [ 460.332779] Lustre: lustre-MDT0001: new disk, initializing [ 460.425322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 460.484861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 460.501972] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 465.112714] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 475.328087] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 481.899955] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 489.454795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 489.878110] Lustre: lustre-OST0000: new disk, initializing [ 489.890361] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 489.899356] Lustre: 17064:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 490.021506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.809441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 491.820600] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 491.900084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 496.289647] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 506.243349] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 512.174346] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 512.326877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 517.621380] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 517.820393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 522.725429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 522.726647] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.732200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 522.769468] Lustre: Skipped 2 previous similar messages [ 526.188407] Lustre: server umount lustre-OST0000 complete [ 531.936065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 531.949846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.986197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 531.989315] Lustre: Skipped 1 previous similar message [ 544.223132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 544.661593] Lustre: server umount lustre-MDT0000 complete [ 547.299947] LustreError: 15104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 547.314872] LustreError: 15104:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 549.021048] LustreError: 15088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825123 with bad export cookie 8263992823009850547 [ 549.022468] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 549.031816] LustreError: 15088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 549.652442] Lustre: server umount lustre-MDT0001 complete [ 554.459308] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 06:05:27 (1787825127) [ 557.492889] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 559.361131] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 06:05:32 (1787825132) [ 566.652239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 566.958564] LustreError: 19476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.022731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 572.076040] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 572.385966] LustreError: 19477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.409396] LustreError: 19477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 581.602077] LustreError: 19476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 581.613309] LustreError: 19476:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 582.162499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 588.262046] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 596.905793] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 604.251652] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 611.831224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 618.822414] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 627.253815] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 647.138655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 647.141958] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.147180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 647.147187] Lustre: Skipped 6 previous similar messages [ 647.184978] Lustre: Skipped 2 previous similar messages [ 651.979334] Lustre: server umount lustre-OST0000 complete [ 670.175212] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 670.418333] Lustre: server umount lustre-MDT0000 complete [ 672.746545] LustreError: 19471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 672.776790] LustreError: 19471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 675.143561] LustreError: 19458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825250 with bad export cookie 8263992823009851429 [ 675.149385] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 675.167211] LustreError: 19458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 675.676315] Lustre: server umount lustre-MDT0001 complete [ 687.829638] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 690.781539] Key type lgssc unregistered [ 691.043154] LNet: 23036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.056471] LNetError: 23036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 692.136189] LNet: Removed LNI 192.168.201.106@tcp [ 693.502499] Key type .llcrypt unregistered [ 693.505791] Key type ._llcrypt unregistered [ 721.301401] Key type ._llcrypt registered [ 721.304461] Key type .llcrypt registered [ 721.404773] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 722.732860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 722.811342] alg: No test for adler32 (adler32-zlib) [ 723.925851] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 724.189972] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 725.943913] Key type lgssc registered [ 727.402547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 737.200667] loop: module loaded [ 737.274372] loop0: detected capacity change from 0 to 204800000 [ 737.585881] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 737.604521] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 738.007133] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 745.283337] loop0: detected capacity change from 0 to 204800000 [ 745.493721] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 745.502901] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 745.530461] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 752.223409] loop0: detected capacity change from 0 to 204800000 [ 758.973654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 759.013120] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 760.424268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 760.494249] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 760.610403] Lustre: lustre-MDT0000: new disk, initializing [ 760.771545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 760.805057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 766.267921] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 776.741466] loop1: detected capacity change from 0 to 204800000 [ 783.076592] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 783.591540] Lustre: lustre-OST07c6: new disk, initializing [ 783.593563] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 783.605720] Lustre: Skipped 1 previous similar message [ 783.626161] Lustre: 26227:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 783.710584] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 790.034229] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 791.200013] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 791.206165] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 791.250074] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 795.974909] Lustre: 25215:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 795.982926] Lustre: 25215:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 795.992693] Lustre: 25215:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 796.001429] Lustre: 25215:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 796.013849] Lustre: 25215:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 796.022716] Lustre: 25215:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 807.837881] Lustre: 25216:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 807.852387] Lustre: 25216:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 807.864952] Lustre: 25216:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 807.873286] Lustre: 25216:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 807.885395] Lustre: 25216:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 807.890389] Lustre: 25216:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 813.342570] Lustre: 25216:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 813.353180] Lustre: 25216:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 813.358502] Lustre: 25216:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 813.363108] Lustre: 25216:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.368447] Lustre: 25216:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 813.373064] Lustre: 25216:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.377626] Lustre: 25216:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 813.384795] Lustre: 25216:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.393143] Lustre: 25216:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 813.401840] Lustre: 25216:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.410169] Lustre: 25216:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 813.425205] Lustre: 25216:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 816.648194] Lustre: 26909:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 816.663723] Lustre: 26909:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 816.667836] Lustre: 26909:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 816.679983] Lustre: 26909:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 816.690825] Lustre: 26909:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 816.706551] Lustre: 26909:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 816.718258] Lustre: 26909:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 816.726987] Lustre: 26909:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 816.734194] Lustre: 26909:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 816.738163] Lustre: 26909:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 816.744428] Lustre: 26909:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 816.751364] Lustre: 26909:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 822.020193] Lustre: 25217:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 822.036123] Lustre: 25217:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 822.045190] Lustre: 25217:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 822.051890] Lustre: 25217:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 822.058614] Lustre: 25217:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 822.070097] Lustre: 25217:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 822.078730] Lustre: 25217:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 822.086643] Lustre: 25217:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 822.095338] Lustre: 25217:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 822.102861] Lustre: 25217:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 822.111173] Lustre: 25217:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 822.121188] Lustre: 25217:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 830.573643] Lustre: 26909:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 830.585897] Lustre: 26909:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 830.603238] Lustre: 26909:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 830.626734] Lustre: 26909:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 830.639605] Lustre: 26909:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 830.659179] Lustre: 26909:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 830.684958] Lustre: 26909:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 830.703107] Lustre: 26909:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 830.713355] Lustre: 26909:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 830.730844] Lustre: 26909:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 830.748358] Lustre: 26909:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 830.757133] Lustre: 26909:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 883.138149] Lustre: server umount lustre-OST07c6 complete [ 902.795397] Lustre: server umount lustre-MDT0000 complete [ 921.010062] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 922.980849] Key type lgssc unregistered [ 923.165804] LNet: 28787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.171937] LNetError: 28787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.183879] LNet: Removed LNI 192.168.201.106@tcp [ 923.781069] Key type .llcrypt unregistered [ 923.782936] Key type ._llcrypt unregistered [ 941.261148] Key type ._llcrypt registered [ 941.264710] Key type .llcrypt registered [ 941.368137] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 942.450156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 942.476267] alg: No test for adler32 (adler32-zlib) [ 943.639475] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 943.855355] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 945.511269] Key type lgssc registered [ 947.168903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 956.917850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 956.956863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 958.735264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 964.306874] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 968.163605] LustreError: 30056:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 968.177278] LustreError: 30056:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 973.289213] LustreError: 30057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 975.016825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 975.551404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 980.266600] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 989.191558] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 996.087078] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1003.586065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1004.095042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1011.920721] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1020.172563] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1028.657299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1045.984153] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1045.998091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1046.012518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1047.402115] Lustre: server umount lustre-OST0000 complete [ 1052.131846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1052.146155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1052.158495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1054.176713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1054.186104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1059.297408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1059.305275] Lustre: Skipped 2 previous similar messages [ 1060.466581] Lustre: server umount lustre-MDT0000 complete [ 1063.404146] LustreError: 30056:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1063.439195] LustreError: 30056:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1064.434417] LustreError: 30037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825639 with bad export cookie 11627725943038897274 [ 1064.457503] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1064.813391] Lustre: server umount lustre-MDT0001 complete [ 1079.477576] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 1082.518530] Key type lgssc unregistered [ 1082.848726] LNet: 33821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1082.865400] LNetError: 33821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1083.878395] LNet: Removed LNI 192.168.201.106@tcp [ 1084.721787] Key type .llcrypt unregistered [ 1084.730341] Key type ._llcrypt unregistered [ 1106.979499] Key type ._llcrypt registered [ 1106.980868] Key type .llcrypt registered [ 1107.083150] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1108.176328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1108.331654] alg: No test for adler32 (adler32-zlib) [ 1109.463348] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 1109.655881] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 1111.303461] Key type lgssc registered [ 1112.835358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1121.984476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1122.032690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1123.672860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1127.924741] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1134.051286] LustreError: 35090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1134.073435] LustreError: 35090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1135.198123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1135.493029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1139.521576] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1147.159210] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1152.473185] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1159.327312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1159.772991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1166.132593] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1169.009130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1174.331303] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1189.862037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1189.872476] Lustre: Skipped 1 previous similar message [ 1189.872770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1193.449270] Lustre: server umount lustre-OST0000 complete [ 1200.096733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1200.100712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1200.111158] Lustre: Skipped 1 previous similar message [ 1200.128279] Lustre: Skipped 2 previous similar messages [ 1203.706241] Lustre: server umount lustre-MDT0000 complete [ 1205.217596] LustreError: 35090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1205.236781] LustreError: 35090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1207.373502] LustreError: 35072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825782 with bad export cookie 14049490175773480450 [ 1207.381317] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1207.389945] LustreError: 35072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1208.017804] Lustre: server umount lustre-MDT0001 complete [ 1220.882380] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 1224.010571] Key type lgssc unregistered [ 1224.278989] LNet: 38648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1224.291178] LNetError: 38648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1224.303668] LNet: Removed LNI 192.168.201.106@tcp [ 1225.092206] Key type .llcrypt unregistered [ 1225.095394] Key type ._llcrypt unregistered [ 1249.531112] Key type ._llcrypt registered [ 1249.532635] Key type .llcrypt registered [ 1249.721711] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1251.179927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1251.331529] alg: No test for adler32 (adler32-zlib) [ 1252.433971] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 1252.864281] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 1254.647463] Key type lgssc registered [ 1256.363279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1264.827165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1264.852535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1266.389418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1273.274247] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1276.898202] LustreError: 40058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1276.914388] LustreError: 40058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1282.023400] LustreError: 40057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1283.454242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1283.936095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1288.174399] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1296.496736] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1302.706164] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1310.105579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1310.586544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1311.609512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1317.600325] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1325.850991] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1339.013951] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.6@tcp (stopping) [ 1339.376145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1342.341230] Lustre: server umount lustre-MDT0000 complete [ 1344.095567] LustreError: 40052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1347.556039] LustreError: 40038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825922 with bad export cookie 11955190063720342823 [ 1347.571934] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1347.584601] LustreError: 40038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1349.213330] LustreError: 41262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1349.232858] LustreError: 41262:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1349.263578] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.6@tcp (stopping) [ 1349.281684] Lustre: Skipped 3 previous similar messages [ 1349.600928] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1349.610473] Lustre: Skipped 2 previous similar messages [ 1352.609619] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1352.611640] Lustre: Skipped 1 previous similar message [ 1354.074701] Lustre: server umount lustre-MDT0001 complete [ 1371.035871] Lustre: server umount lustre-OST0000 complete [ 1390.221219] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 1393.506292] Key type lgssc unregistered [ 1393.858335] LNet: 43926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1393.869673] LNetError: 43926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1394.926208] LNet: Removed LNI 192.168.201.106@tcp [ 1396.017388] Key type .llcrypt unregistered [ 1396.020059] Key type ._llcrypt unregistered [ 1424.166591] Key type ._llcrypt registered [ 1424.168730] Key type .llcrypt registered [ 1424.245585] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1425.544637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1425.568780] alg: No test for adler32 (adler32-zlib) [ 1426.690118] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 1426.870473] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 1428.530525] Key type lgssc registered [ 1429.872771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1437.102466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1437.115532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1438.649901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1443.159168] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1448.934930] LustreError: 45344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1448.952208] LustreError: 45344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1453.045428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1453.528082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1457.466891] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1464.314699] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1470.094862] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1476.067363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1476.391475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1481.847113] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1483.594914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1489.450807] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1496.214703] Lustre: lustre-OST0000: Not available for connect from 192.168.201.6@tcp (stopping) [ 1499.106072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1499.128963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1499.156872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1499.167432] Lustre: Skipped 1 previous similar message [ 1501.290866] Lustre: lustre-OST0000: Not available for connect from 192.168.201.6@tcp (stopping) [ 1501.463542] Lustre: server umount lustre-OST0000 complete [ 1524.706554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1524.713869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1524.721028] Lustre: Skipped 2 previous similar messages [ 1524.738988] Lustre: Skipped 1 previous similar message [ 1527.124125] Lustre: server umount lustre-MDT0000 complete [ 1529.831784] LustreError: 45345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1529.860204] LustreError: 45345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1530.821461] LustreError: 45324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826105 with bad export cookie 3935038207806902816 [ 1530.824113] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1530.835688] LustreError: 45324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1545.183243] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1545.931817] Lustre: server umount lustre-MDT0001 complete [ 1563.307045] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 1567.286802] Key type lgssc unregistered [ 1567.773187] LNet: 49080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1567.781775] LNetError: 49080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1567.803226] LNet: Removed LNI 192.168.201.106@tcp [ 1568.738566] Key type .llcrypt unregistered [ 1568.742659] Key type ._llcrypt unregistered [ 1598.504391] Key type ._llcrypt registered [ 1598.506276] Key type .llcrypt registered [ 1598.591754] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1599.786092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1599.821542] alg: No test for adler32 (adler32-zlib) [ 1600.831663] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 1601.106793] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 1602.839306] Key type lgssc registered [ 1603.910875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1611.490963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1611.540352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1612.959791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1617.558858] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1622.512436] LustreError: 50498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1622.532038] LustreError: 50498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1625.648041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1625.888534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1630.275519] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1638.108543] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1643.345653] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1650.022356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1650.271950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1656.667442] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1657.463258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1665.310326] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1671.443687] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1674.703806] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1687.449637] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-27 6h24m19s [ 1689.297042] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1692.645279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1692.655209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1692.668486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1693.728718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1693.746511] Lustre: Skipped 4 previous similar messages [ 1694.879412] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.6@tcp (stopping) [ 1694.893665] Lustre: Skipped 1 previous similar message [ 1697.943930] Lustre: server umount lustre-MDT0000 complete [ 1699.810959] LustreError: 50498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1699.824860] LustreError: 50498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1701.515455] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1705.953100] LustreError: 50497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1705.990058] LustreError: 50497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1708.917961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1709.133139] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1709.592695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1709.655453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1710.635092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1715.135915] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1719.269465] Lustre: DEBUG MARKER: Wait for df (36708) ... [ 1720.846855] Lustre: DEBUG MARKER: done [ 1725.923350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1725.939307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1725.950795] Lustre: Skipped 3 previous similar messages [ 1731.045909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1731.053538] Lustre: Skipped 2 previous similar messages [ 1740.265820] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1740.529798] Lustre: server umount lustre-OST0000 complete [ 1746.402755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.411938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1746.420472] Lustre: Skipped 3 previous similar messages [ 1750.950809] Lustre: server umount lustre-MDT0000 complete [ 1753.571651] LustreError: 50498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1753.584542] LustreError: 50498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1755.572567] LustreError: 54678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826330 with bad export cookie 7266621975962156725 [ 1755.588878] LustreError: 54678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1755.590217] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1756.193685] Lustre: server umount lustre-MDT0001 complete [ 1767.115921] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 1770.429149] Key type lgssc unregistered [ 1770.759104] LNet: 55433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1770.764838] LNetError: 55433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1771.813791] LNet: Removed LNI 192.168.201.106@tcp [ 1772.637820] Key type .llcrypt unregistered [ 1772.641854] Key type ._llcrypt unregistered [ 1816.990639] Key type ._llcrypt registered [ 1816.992601] Key type .llcrypt registered [ 1817.083579] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1818.188962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1818.214628] alg: No test for adler32 (adler32-zlib) [ 1819.344489] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 1819.564601] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 1821.199437] Key type lgssc registered [ 1822.425127] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1831.286227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1831.323119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1832.663284] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1832.720722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1832.996880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1837.768470] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1847.196342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1847.308346] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1847.343041] Lustre: 57643:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1847.641746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1852.366511] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1861.261129] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1868.677836] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1876.711641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1877.005213] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1877.228717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1884.980726] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1887.478766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1894.299241] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1900.773941] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-27 6h27m53s [ 1902.684239] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1912.469650] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1916.484371] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1916.554951] LustreError: 57651:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1937.065752] Lustre: lustre-MDT0000: Client dfb1c516-7c5d-42b9-84fd-0e502cc37125 (at 192.168.201.6@tcp) reconnecting [ 1956.591183] LustreError: 57651:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1958.773140] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1961.101331] Lustre: DEBUG MARKER: done [ 1969.088040] Lustre: server umount lustre-OST0000 complete [ 1974.753422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1974.773512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1974.783831] Lustre: Skipped 1 previous similar message [ 1975.852935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1981.924961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1981.932307] Lustre: Skipped 2 previous similar messages [ 1984.160213] Lustre: server umount lustre-MDT0000 complete [ 1986.020406] Lustre: 56965:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826544/real 1787826544] req@00000000a6b5aee2 x1874671848972160/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787826560 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1986.052765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1986.068208] Lustre: Skipped 1 previous similar message [ 1988.540586] LustreError: 57637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826563 with bad export cookie 12797908400314096216 [ 1988.549405] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1988.563946] LustreError: 57637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1989.056716] Lustre: server umount lustre-MDT0001 complete [ 2007.393086] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 2011.434922] Key type lgssc unregistered [ 2011.959477] LNet: 61999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2011.963334] LNetError: 61999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2013.029725] LNet: Removed LNI 192.168.201.106@tcp [ 2014.321866] Key type .llcrypt unregistered [ 2014.324750] Key type ._llcrypt unregistered [ 2070.621859] Key type ._llcrypt registered [ 2070.627720] Key type .llcrypt registered [ 2070.812126] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2072.886992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2072.967671] alg: No test for adler32 (adler32-zlib) [ 2074.174963] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 2074.608764] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 2076.360329] Key type lgssc registered [ 2078.350402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2082.354974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2082.403782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2083.933843] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2083.995145] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2084.177049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2085.312034] Lustre: Failing over lustre-MDT0000 [ 2085.830120] Lustre: server umount lustre-MDT0000 complete [ 2094.834978] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 06:31:07 (1787826667) [ 2103.537451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2104.212985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2109.487514] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2120.281924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2120.408160] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2120.463234] Lustre: 64810:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2120.773894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2126.135660] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2135.859510] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2142.659601] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2150.295635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2150.689295] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2150.988786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2157.444485] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2158.209307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2165.295228] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2171.884961] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2179.041128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2179.046519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2179.054368] Lustre: Skipped 1 previous similar message [ 2179.063537] Lustre: Skipped 2 previous similar messages [ 2184.163403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2184.172541] Lustre: Skipped 2 previous similar messages [ 2189.283206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2189.288522] Lustre: Skipped 2 previous similar messages [ 2193.378693] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2193.582985] Lustre: server umount lustre-MDT0000 complete [ 2198.835598] LustreError: 66668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826773 with bad export cookie 16460574300778956698 [ 2198.842603] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2198.851367] LustreError: 66668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2199.362181] Lustre: server umount lustre-MDT0001 complete [ 2204.118657] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2211.448710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2215.903278] Lustre: 63731:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826774/real 1787826774] req@00000000c308a78a x1874672116345344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787826790 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2215.956566] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2215.976701] Lustre: Skipped 1 previous similar message [ 2219.681078] Lustre: 63732:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826778/real 1787826778] req@00000000f815af6e x1874672116345472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787826794 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2221.028776] Lustre: 63729:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826779/real 1787826779] req@000000009750aa42 x1874672116345600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787826795 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2223.668259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2224.159546] Lustre: 63729:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826783/real 1787826783] req@00000000fea236e7 x1874672116345856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787826799 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2229.216790] Lustre: 63731:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826788/real 1787826788] req@000000002ea0c222 x1874672116345984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787826804 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2243.561872] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe46fb51d5cda907d [ 2243.592552] Lustre: MGC192.168.201.106@tcp: Connection restored to 0@lo (at 0@lo) [ 2243.996879] LustreError: 66662:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2244.029171] LustreError: 66662:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2244.148239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2244.268752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2245.171456] LustreError: 66662:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2245.206771] LustreError: 66662:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2249.154594] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2249.260232] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2249.257855] LustreError: 66663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2254.310217] LustreError: 66662:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2257.494129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2257.748593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2260.850067] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2262.345342] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2271.073691] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2278.294879] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2288.612630] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2288.629525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2288.651200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2291.681602] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2291.690178] Lustre: Skipped 1 previous similar message [ 2293.648113] Lustre: server umount lustre-MDT0000 complete [ 2296.801325] LustreError: 68589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2296.816226] LustreError: 68589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2297.333845] LustreError: 64805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826872 with bad export cookie 16460574300778958973 [ 2297.336137] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2297.359640] LustreError: 64805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2297.686876] Lustre: server umount lustre-MDT0001 complete [ 2305.182682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2315.898903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2317.279612] Lustre: 63732:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826876/real 1787826876] req@000000002dc3bc8f x1874672116375552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787826892 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2317.320057] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2322.821350] LustreError: 66663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2322.937984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2322.988644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2327.354788] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2336.915870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2341.404372] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2342.913860] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2347.050683] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2351.367604] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2360.971244] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2372.577466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2372.582313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2372.600419] Lustre: Skipped 1 previous similar message [ 2372.611635] Lustre: Skipped 5 previous similar messages [ 2375.571402] Lustre: server umount lustre-MDT0000 complete [ 2377.702514] LustreError: 71271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2377.738590] LustreError: 71271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2380.850835] LustreError: 66668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826955 with bad export cookie 16460574300778960352 [ 2380.853616] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2380.856302] LustreError: 66668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2381.492682] Lustre: server umount lustre-MDT0001 complete [ 2388.476106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2391.959971] Lustre: DEBUG MARKER: files compared the same [ 2404.136692] Lustre: server umount lustre-OST0000 complete [ 2427.559657] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 2430.742756] Key type lgssc unregistered [ 2431.090687] LNet: 74555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2431.095914] LNetError: 74555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2431.115694] LNet: Removed LNI 192.168.201.106@tcp [ 2432.093663] Key type .llcrypt unregistered [ 2432.095419] Key type ._llcrypt unregistered [ 2459.600479] Key type ._llcrypt registered [ 2459.602509] Key type .llcrypt registered [ 2459.729683] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2461.146378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2461.202286] alg: No test for adler32 (adler32-zlib) [ 2462.348816] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 2462.742059] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 2464.447191] Key type lgssc registered [ 2465.727796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2473.970553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2474.004869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2475.541301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2480.359723] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug +malloc all [ 2485.729419] LustreError: 75973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2485.743739] LustreError: 75973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2490.309651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2490.707512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2495.566473] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug +malloc all [ 2504.814742] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2513.724602] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2521.290450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2521.572573] LustreError: 77795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2521.733414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2527.214685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2530.516981] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug +malloc all [ 2538.876110] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2551.780119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2551.781821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2551.790589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2555.939372] Lustre: server umount lustre-OST0000 complete [ 2562.015893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2562.036694] Lustre: Skipped 1 previous similar message [ 2562.045485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2562.056777] Lustre: Skipped 2 previous similar messages [ 2567.139038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2567.150894] Lustre: Skipped 1 previous similar message [ 2567.444211] Lustre: server umount lustre-MDT0000 complete [ 2571.134819] LustreError: 75955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827146 with bad export cookie 8995866994653721961 [ 2571.142605] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2571.152324] LustreError: 75955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2571.551487] Lustre: server umount lustre-MDT0001 complete [ 2585.181563] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 2588.327970] Key type lgssc unregistered [ 2588.709594] LNet: 79511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2588.724668] LNetError: 79511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2588.767731] LNet: Removed LNI 192.168.201.106@tcp [ 2589.768279] Key type .llcrypt unregistered [ 2589.771399] Key type ._llcrypt unregistered [ 2620.091578] Key type ._llcrypt registered [ 2620.100370] Key type .llcrypt registered [ 2620.294846] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2621.507783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2621.588366] alg: No test for adler32 (adler32-zlib) [ 2622.659324] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 2622.938852] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 2624.687199] Key type lgssc registered [ 2625.951730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2635.417822] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2635.455183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2651.105663] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2656.223488] LustreError: 80902:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.106@tcp: failed processing log, type 4: rc = -110 [ 2682.914169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2690.295443] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2699.306944] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2708.562783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2708.719153] Lustre: *** cfs_fail_loc=706, val=0*** [ 2709.242452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2709.313909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2713.856308] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2714.600594] LustreError: 80928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2714.614434] LustreError: 80928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2717.678753] LustreError: 81987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2717.715283] LustreError: 81987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2722.784984] LustreError: 81988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2722.817854] LustreError: 81988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2723.146702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2723.544730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2728.434955] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2737.861676] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2744.419560] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2749.418978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2749.431610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2749.436864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2749.442792] Lustre: Skipped 1 previous similar message [ 2752.870837] Lustre: server umount lustre-OST0000 complete [ 2759.653795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2759.672665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2759.679069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2759.690822] Lustre: Skipped 2 previous similar messages [ 2759.713296] Lustre: Skipped 1 previous similar message [ 2763.877190] Lustre: server umount lustre-MDT0000 complete [ 2764.768227] LustreError: 81993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2764.804251] LustreError: 81993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2768.328119] LustreError: 80910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827343 with bad export cookie 6344409168460925804 [ 2768.364634] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2768.880497] Lustre: server umount lustre-MDT0001 complete [ 2786.767787] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 2790.674491] Key type lgssc unregistered [ 2791.181724] LNet: 84543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2791.202359] LNetError: 84543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2791.240885] LNet: Removed LNI 192.168.201.106@tcp [ 2791.995216] Key type .llcrypt unregistered [ 2791.997609] Key type ._llcrypt unregistered [ 2826.839894] Key type ._llcrypt registered [ 2826.845204] Key type .llcrypt registered [ 2827.011484] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2828.882433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2828.898472] alg: No test for adler32 (adler32-zlib) [ 2830.064700] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 2830.423321] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 2832.143165] Key type lgssc registered [ 2833.381930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2842.389234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2842.406655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2849.632611] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2858.557982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2859.200136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2864.299798] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2864.608408] LustreError: 86652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2864.616648] LustreError: 86652:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2868.708046] LustreError: 86655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2868.732050] LustreError: 86655:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2873.632880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2873.825484] LustreError: 86656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2873.852985] LustreError: 86656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2874.030805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2879.472132] LustreError: 86655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2879.501674] LustreError: 86655:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2881.743042] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2883.559684] LustreError: 86656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2883.579276] LustreError: 86656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2891.965428] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2891.974982] Lustre: MGS: Client 4086353f-9c78-4b25-8279-b2c4da6eaea1 (at 0@lo) reconnecting [ 2892.015093] LustreError: 86666:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2892.200755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2892.283438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2897.333480] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2912.742671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2912.759208] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2912.766252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2912.769714] Lustre: Skipped 1 previous similar message [ 2917.861096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2917.872948] Lustre: Skipped 1 previous similar message [ 2918.767149] Lustre: server umount lustre-OST0000 complete [ 2928.096821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2928.103462] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2928.114610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2928.127016] Lustre: Skipped 1 previous similar message [ 2928.152530] Lustre: Skipped 2 previous similar messages [ 2929.820469] Lustre: server umount lustre-MDT0000 complete [ 2933.217939] LustreError: 86656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2933.247381] LustreError: 86656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2935.159356] Lustre: server umount lustre-MDT0001 complete [ 2940.041761] Lustre: server umount MGS complete [ 2957.286465] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 06:45:29 (1787827529) [ 2996.888172] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 3009.027885] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3049.785975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3054.245120] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3064.571428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3064.773902] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3064.829254] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3064.896554] Lustre: lustre-MDT0001: new disk, initializing [ 3064.956876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3064.980662] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3064.989622] Lustre: 93845:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3065.004285] LustreError: 93845:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3070.263787] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3083.614521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3084.158362] Lustre: lustre-OST0000: new disk, initializing [ 3084.164489] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3084.180439] Lustre: 94614:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3084.308835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3091.884895] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3094.115604] Lustre: 95006:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000359f55ef x1874672909067904/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787827685 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3102.765749] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3109.733753] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3109.742804] Lustre: MGS: Client 874c4247-46e9-4ae7-9dc8-1711627065cd (at 0@lo) reconnecting [ 3109.772691] Lustre: 93115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3109.787927] LustreError: 93831:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3109.803301] LustreError: 93831:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3109.817211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3109.870569] Lustre: lustre-MDT0000: new disk, initializing [ 3110.001465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3110.023495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3110.037727] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3111.077222] Lustre: 95683:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000f7077147 x1874672909074816/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787827701 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3114.653845] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3116.212736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3116.218631] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3116.257798] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3119.213992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 3119.223790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 3142.117856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3142.139334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3142.145086] Lustre: Skipped 1 previous similar message [ 3142.153075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3142.161610] Lustre: Skipped 1 previous similar message [ 3142.183353] LustreError: Skipped 1 previous similar message [ 3147.292597] Lustre: server umount lustre-OST0000 complete [ 3152.354113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3152.371073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3152.383579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3152.396215] Lustre: Skipped 1 previous similar message [ 3157.529543] Lustre: server umount lustre-MDT0000 complete [ 3167.948894] Lustre: server umount lustre-MDT0001 complete [ 3172.544819] Lustre: server umount MGS complete [ 3184.995770] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 06:49:17 (1787827757) [ 3205.108404] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 3207.934962] Key type lgssc unregistered [ 3208.237134] LNet: 98182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3208.248429] LNetError: 98182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3208.273038] LNet: Removed LNI 192.168.201.106@tcp [ 3209.073220] Key type .llcrypt unregistered [ 3209.075574] Key type ._llcrypt unregistered [ 3228.852701] Key type ._llcrypt registered [ 3228.854311] Key type .llcrypt registered [ 3229.001986] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3230.457709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3230.536760] alg: No test for adler32 (adler32-zlib) [ 3231.581592] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 3231.763322] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 3233.376546] Key type lgssc registered [ 3234.774536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3242.257236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3242.300936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3243.815710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3248.278640] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3253.232429] LustreError: 99404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3253.258394] LustreError: 99404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3257.411846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3257.798629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3262.479509] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3270.875290] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3277.910603] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3284.757109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3285.088194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3289.234140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 3291.575251] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3300.611827] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3315.170154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3315.185169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3315.215087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3318.251765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3318.252128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3318.289714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3320.015693] Lustre: server umount lustre-OST0000 complete [ 3324.386789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3324.405204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3324.427150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3324.431977] Lustre: Skipped 1 previous similar message [ 3328.492775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3328.502889] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3330.787513] Lustre: server umount lustre-MDT0000 complete [ 3333.601732] LustreError: 99405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3333.654351] LustreError: 99405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3335.862344] LustreError: 99385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827910 with bad export cookie 13370965576524297173 [ 3335.880308] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3335.884632] LustreError: 99385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3336.605305] Lustre: server umount lustre-MDT0001 complete [ 3353.310538] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 3356.268428] Key type lgssc unregistered [ 3356.583754] LNet: 102982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3356.594414] LNetError: 102982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3356.609344] LNet: Removed LNI 192.168.201.106@tcp [ 3357.449875] Key type .llcrypt unregistered [ 3357.454903] Key type ._llcrypt unregistered [ 3377.374659] Key type ._llcrypt registered [ 3377.378475] Key type .llcrypt registered [ 3377.553632] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3378.984704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3379.053504] alg: No test for adler32 (adler32-zlib) [ 3380.212175] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 3380.580675] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 3382.247173] Key type lgssc registered [ 3383.626274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3389.999923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3390.015327] LustreError: 104057:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3390.074137] LustreError: 104064:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3390.085722] LustreError: 104057:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4940 [ 3390.095196] LustreError: 104057:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3390.100948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3390.104400] LustreError: 104057:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3391.759898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3400.394640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3400.722763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3405.434795] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3410.579670] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing lsmod [ 3413.431964] LustreError: 105265:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3413.463726] LustreError: 105271:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3413.469914] LustreError: 105265:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4970 [ 3413.480164] LustreError: 105265:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3413.489076] LustreError: 105265:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3413.521430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3413.912914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3419.115722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3419.119369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3419.135222] Lustre: Skipped 2 previous similar messages [ 3419.152452] Lustre: Skipped 2 previous similar messages [ 3422.906615] Lustre: server umount lustre-MDT0000 complete [ 3424.228339] LustreError: 104096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3424.249110] LustreError: 104096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3425.250869] LustreError: 104096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3425.278188] LustreError: 104096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3426.361957] LustreError: 104082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828001 with bad export cookie 14471503008760214202 [ 3426.373092] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3426.379597] LustreError: 104082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3427.361945] LustreError: 104532:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000046d420a0 x1874673485762432/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3427.567825] Lustre: server umount lustre-MDT0001 complete [ 3440.415078] Lustre: server umount lustre-OST0000 complete [ 3448.914700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3449.440027] LustreError: 106336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3449.547275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3454.041954] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3454.946120] LustreError: 106337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3460.069578] LustreError: 106336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3462.866507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3463.128379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3467.639261] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3475.181822] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3480.871801] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3487.861649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3488.158732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3494.199879] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3494.320808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 3502.109620] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3512.799956] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3512.808440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3512.819858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3512.844509] Lustre: Skipped 1 previous similar message [ 3514.795206] Lustre: server umount lustre-OST0000 complete [ 3519.456901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3519.461984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3519.469127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3519.480062] Lustre: Skipped 1 previous similar message [ 3523.042567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3523.045788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3525.772736] Lustre: server umount lustre-MDT0000 complete [ 3528.160417] LustreError: 108825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3528.166963] LustreError: 108825:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3530.272234] LustreError: 106316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828105 with bad export cookie 14471503008760214902 [ 3530.284907] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3530.652307] Lustre: server umount lustre-MDT0001 complete [ 3542.431321] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 3545.061517] Key type lgssc unregistered [ 3545.369799] LNet: 109869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3545.377579] LNetError: 109869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3545.418332] LNet: Removed LNI 192.168.201.106@tcp [ 3546.218180] Key type .llcrypt unregistered [ 3546.219714] Key type ._llcrypt unregistered [ 3569.543801] Key type ._llcrypt registered [ 3569.545929] Key type .llcrypt registered [ 3569.648908] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3570.932487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3570.955558] alg: No test for adler32 (adler32-zlib) [ 3572.057764] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 3572.308694] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 3574.040081] Key type lgssc registered [ 3575.254055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3582.350805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3582.384331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3583.792553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3588.045716] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3593.197500] LustreError: 111285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3593.217415] LustreError: 111285:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3597.052051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3597.635781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3602.024228] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3610.800425] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3617.395207] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3626.123322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3626.684740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3628.715253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 3635.537671] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3645.925312] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3651.616108] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3655.426772] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3663.856527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3663.864579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3663.880829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3666.630218] Lustre: server umount lustre-OST0000 complete [ 3674.093677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3674.104632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3674.127148] Lustre: Skipped 2 previous similar messages [ 3674.143339] Lustre: Skipped 2 previous similar messages [ 3679.204087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3679.216909] Lustre: Skipped 1 previous similar message [ 3683.298259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3683.307202] Lustre: Skipped 1 previous similar message [ 3685.345685] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3685.584486] Lustre: server umount lustre-MDT0000 complete [ 3688.427240] LustreError: 112524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3688.453820] LustreError: 112524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3690.242242] LustreError: 111266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828265 with bad export cookie 11055831651973657164 [ 3690.247950] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3690.250060] LustreError: 111266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3690.641530] Lustre: server umount lustre-MDT0001 complete [ 3704.944339] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 3707.752538] Key type lgssc unregistered [ 3707.981627] LNet: 114930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3707.995762] LNetError: 114930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3708.019403] LNet: Removed LNI 192.168.201.106@tcp [ 3708.925072] Key type .llcrypt unregistered [ 3708.926777] Key type ._llcrypt unregistered [ 3727.078637] Key type ._llcrypt registered [ 3727.083694] Key type .llcrypt registered [ 3727.183847] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3728.478364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3728.529665] alg: No test for adler32 (adler32-zlib) [ 3729.707325] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 3729.972393] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 3731.623265] Key type lgssc registered [ 3732.976064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3741.874668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3741.910912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3743.684226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3749.789358] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3753.985239] LustreError: 116154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3754.015946] LustreError: 116154:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3759.077062] LustreError: 116153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3760.056523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3760.507421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3766.790686] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3776.821880] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3784.316084] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3791.992416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3792.430849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3799.553504] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3808.251800] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3813.954479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3824.098872] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3824.103150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3824.119330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3824.127754] Lustre: Skipped 1 previous similar message [ 3824.158280] Lustre: Skipped 1 previous similar message [ 3829.219561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3829.242376] Lustre: Skipped 1 previous similar message [ 3830.339179] Lustre: server umount lustre-OST0000 complete [ 3837.409087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3837.433106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3837.467165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3839.457233] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3839.479366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3839.487038] Lustre: Skipped 1 previous similar message [ 3844.580186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3844.588218] Lustre: Skipped 1 previous similar message [ 3849.183693] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3849.598462] Lustre: server umount lustre-MDT0000 complete [ 3849.709863] LustreError: 116154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3849.738127] LustreError: 116154:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3854.132157] LustreError: 116135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828429 with bad export cookie 2888864902784919676 [ 3854.145654] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3854.810508] Lustre: server umount lustre-MDT0001 complete [ 3867.404136] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 3871.328818] Key type lgssc unregistered [ 3871.708791] LNet: 119714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3871.719724] LNetError: 119714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3871.771901] LNet: Removed LNI 192.168.201.106@tcp [ 3873.374568] Key type .llcrypt unregistered [ 3873.382766] Key type ._llcrypt unregistered [ 3903.406051] Key type ._llcrypt registered [ 3903.412725] Key type .llcrypt registered [ 3903.519700] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3904.682369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3904.804070] alg: No test for adler32 (adler32-zlib) [ 3906.006650] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 3906.282046] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 3908.007467] Key type lgssc registered [ 3909.278240] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3917.843737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3917.871756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3919.439420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3925.871303] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3929.573858] LustreError: 121148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3929.617498] LustreError: 121148:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3934.692504] LustreError: 121149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3936.837242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3937.185305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3942.011676] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3950.607833] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3957.497940] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3965.331654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3965.962297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3972.734699] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3980.667906] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3989.510183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3998.398627] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4002.199637] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4002.205237] Lustre: Skipped 2 previous similar messages [ 4007.193231] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4007.208699] Lustre: Skipped 2 previous similar messages [ 4007.514769] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4007.562376] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4018.694791] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.6@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4018.701350] Lustre: Skipped 2 previous similar messages [ 4028.056065] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.6@tcp 192.168.0.[2,10]@tcp [ 4036.744404] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.6/24@tcp in log lustre-MDT0000 [ 4036.767865] Lustre: Skipped 2 previous similar messages [ 4042.907846] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.6/24@tcp [ 4042.926266] Lustre: Skipped 1 previous similar message [ 4056.034671] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4056.043494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4056.064908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4056.117843] Lustre: Skipped 1 previous similar message [ 4060.036289] Lustre: server umount lustre-OST0000 complete [ 4066.282071] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4066.283974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4066.300089] Lustre: Skipped 1 previous similar message [ 4066.302790] Lustre: Skipped 1 previous similar message [ 4071.402266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4071.410776] Lustre: Skipped 2 previous similar messages [ 4072.385902] Lustre: server umount lustre-MDT0000 complete [ 4076.514271] LustreError: 121145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4076.536635] LustreError: 121145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4077.350080] LustreError: 121129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828652 with bad export cookie 11812143291948011476 [ 4077.360806] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4077.371890] LustreError: 121129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4077.923566] Lustre: server umount lustre-MDT0001 complete [ 4090.221514] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 4094.031561] Key type lgssc unregistered [ 4094.387182] LNet: 126115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4094.401741] LNetError: 126115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4094.443105] LNet: Removed LNI 192.168.201.106@tcp [ 4095.789391] Key type .llcrypt unregistered [ 4095.792193] Key type ._llcrypt unregistered [ 4125.706591] Key type ._llcrypt registered [ 4125.708961] Key type .llcrypt registered [ 4125.808215] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 4126.846854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4127.175550] alg: No test for adler32 (adler32-zlib) [ 4128.183284] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 4128.465576] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 4130.231622] Key type lgssc registered [ 4131.430424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4139.871798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4139.917536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4141.395120] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4141.421177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4141.429281] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.6/24@tcp [ 4146.191365] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4151.776537] LustreError: 127687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4151.794061] LustreError: 127687:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4155.509488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4155.788058] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4155.821690] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.6/24@tcp [ 4155.878646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4160.015563] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4168.084346] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4174.988577] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4183.383135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4183.863851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4186.962712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 4191.334887] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4199.282424] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4216.805557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4216.809480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4216.810252] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4216.825552] Lustre: Skipped 1 previous similar message [ 4216.863628] Lustre: Skipped 1 previous similar message [ 4219.041620] Lustre: server umount lustre-OST0000 complete [ 4227.044780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4227.075080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4227.083227] Lustre: Skipped 1 previous similar message [ 4229.739602] Lustre: server umount lustre-MDT0000 complete [ 4232.163173] LustreError: 127682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4232.194905] LustreError: 127682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4233.800262] LustreError: 127668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828808 with bad export cookie 13479499277002196309 [ 4233.805188] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4233.829487] LustreError: 127668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4234.275193] Lustre: server umount lustre-MDT0001 complete [ 4247.998191] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 4251.707910] Key type lgssc unregistered [ 4252.130276] LNet: 131281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4252.145269] LNetError: 131281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4253.223443] LNet: Removed LNI 192.168.201.106@tcp [ 4254.212336] Key type .llcrypt unregistered [ 4254.216324] Key type ._llcrypt unregistered [ 4287.893665] Key type ._llcrypt registered [ 4287.895799] Key type .llcrypt registered [ 4288.026526] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 4311.028307] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 4313.384814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4313.411419] alg: No test for adler32 (adler32-zlib) [ 4314.590824] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 4315.014518] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 4316.783275] Key type lgssc registered [ 4318.321113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4363.975075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4364.005857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4365.400769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4365.443157] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4365.562955] Lustre: lustre-MDT0000: new disk, initializing [ 4365.644894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4365.662782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4370.005165] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4382.255982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4382.513488] Lustre: 134687:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4382.627594] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4382.632938] Lustre: Skipped 1 previous similar message [ 4382.746378] Lustre: lustre-MDT0001: new disk, initializing [ 4382.841723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4382.902926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4382.916864] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4388.564218] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4400.923212] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4407.420603] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4415.855404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4416.132761] Lustre: lustre-OST0000: new disk, initializing [ 4416.136536] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4416.146387] Lustre: 136658:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4416.296920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4423.068360] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4423.210463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4423.213574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4423.419642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4434.278323] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4440.130554] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4440.367546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4445.207868] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4445.411789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4448.738364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4448.750181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4448.771493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4449.255440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.257540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4449.277448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4449.285244] Lustre: Skipped 1 previous similar message [ 4453.559486] Lustre: server umount lustre-OST0000 complete [ 4459.512991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4459.519344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4459.520218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4459.528493] Lustre: Skipped 1 previous similar message [ 4459.543278] Lustre: Skipped 1 previous similar message [ 4464.608740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4464.618974] Lustre: Skipped 1 previous similar message [ 4469.731974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4469.749855] Lustre: Skipped 1 previous similar message [ 4472.287153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4472.571885] Lustre: server umount lustre-MDT0000 complete [ 4476.662539] LustreError: 134678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829051 with bad export cookie 13079859488980151110 [ 4476.665321] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4476.685450] LustreError: 134678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4477.084651] Lustre: server umount lustre-MDT0001 complete [ 4486.384168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4487.042700] LustreError: 138875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4487.069548] LustreError: 138875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4487.174271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4492.256409] LustreError: 138876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4492.682262] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4496.357878] LustreError: 138875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4501.472810] LustreError: 138876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4502.610776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4503.256600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4509.193848] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4518.913860] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4528.765689] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4540.723385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4541.113269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4548.468704] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4557.693577] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4563.241563] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4563.489180] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4568.137288] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4568.364627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4577.092250] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4577.277448] Lustre: lustre-OST0001: new disk, initializing [ 4577.280577] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4577.290192] Lustre: 142249:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4577.385291] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4583.382848] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4584.580234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4584.595463] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4584.700405] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4593.191242] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4593.395557] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4598.393958] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4598.715500] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4604.738923] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f7710c56000.ost_server_uuid 50 [ 4606.770346] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f7710c56000.ost_server_uuid in IDLE state after 0 sec [ 4620.769838] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4620.775267] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4620.786110] LustreError: Skipped 1 previous similar message [ 4620.798223] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4620.802700] Lustre: Skipped 1 previous similar message [ 4620.823597] Lustre: Skipped 1 previous similar message [ 4622.031791] Lustre: server umount lustre-OST0001 complete [ 4625.890256] LustreError: 140697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4625.906340] LustreError: 140697:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4628.451730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4628.465596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4632.880683] Lustre: server umount lustre-OST0000 complete [ 4641.249347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4641.250048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4641.253038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4641.253048] LustreError: Skipped 1 previous similar message [ 4641.266887] Lustre: Skipped 2 previous similar messages [ 4641.268937] Lustre: Skipped 4 previous similar messages [ 4645.365655] Lustre: server umount lustre-MDT0000 complete [ 4646.368832] LustreError: 139603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4646.392029] LustreError: 139603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4650.122893] LustreError: 138856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829225 with bad export cookie 13079859488980151992 [ 4650.123882] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4650.133350] LustreError: 138856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4651.126483] Lustre: server umount lustre-MDT0001 complete [ 4679.225751] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 4682.215939] Key type lgssc unregistered [ 4682.577443] LNet: 145260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4682.581076] LNetError: 145260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4682.607071] LNet: Removed LNI 192.168.201.106@tcp [ 4683.585453] Key type .llcrypt unregistered [ 4683.586817] Key type ._llcrypt unregistered [ 4728.279229] Key type ._llcrypt registered [ 4728.281076] Key type .llcrypt registered [ 4728.387479] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 4744.997066] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 4746.780148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4746.835248] alg: No test for adler32 (adler32-zlib) [ 4747.951879] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 4748.276027] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 4749.919161] Key type lgssc registered [ 4751.249699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4789.517784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4789.544779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4790.939556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4790.976600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4791.119413] Lustre: lustre-MDT0000: new disk, initializing [ 4791.254780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4791.287397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4796.336725] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4807.541379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4807.742438] Lustre: 149419:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4807.793615] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4807.796824] Lustre: Skipped 1 previous similar message [ 4807.950863] Lustre: lustre-MDT0001: new disk, initializing [ 4808.068124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4808.113093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4808.132231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4813.809882] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4825.664347] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4832.515911] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4839.356742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4839.624940] Lustre: lustre-OST0000: new disk, initializing [ 4839.638163] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4839.646084] Lustre: 151387:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4839.727663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4845.203674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4845.223103] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4845.401322] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4847.277196] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4859.720790] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4864.509177] Lustre: Failing over lustre-OST0000 [ 4864.887990] Lustre: server umount lustre-OST0000 complete [ 4881.378602] Lustre: 147639:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787829440/real 1787829440] req@00000000ac9818da x1874674920230272/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787829456 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4881.386237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4881.406253] Lustre: 147639:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4882.400746] Lustre: 147640:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787829440/real 1787829440] req@000000006daeb849 x1874674920229248/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787829456 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4882.432787] Lustre: 147640:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4882.440519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4884.207856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4884.532731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4884.554486] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4887.583518] Lustre: 147641:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787829446/real 1787829446] req@0000000023f109db x1874674920231424/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787829462 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4891.403358] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4891.687094] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4892.817379] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4892.817567] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4893.855506] Lustre: 147641:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787829451/real 1787829451] req@000000007f411b46 x1874674920232576/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787829467 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4893.907024] Lustre: 147641:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4895.813225] Lustre: Failing over lustre-MDT0000 [ 4896.223738] Lustre: server umount lustre-MDT0000 complete [ 4896.803683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4896.817433] Lustre: Skipped 1 previous similar message [ 4897.891094] Lustre: 147642:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787829456/real 1787829456] req@00000000389119e2 x1874674920233600/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787829472 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4897.919468] Lustre: 147642:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4901.862799] LustreError: 149430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4901.883330] LustreError: 149430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4903.005823] LustreError: 153051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4906.977857] LustreError: 149429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4906.990899] LustreError: 149429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4912.095400] Lustre: 147639:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787829471/real 1787829471] req@00000000f3952d65 x1874674920244736/t0(0) o400->MGC192.168.201.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787829487 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4912.103996] LustreError: 150580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4912.124286] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4912.140589] LustreError: 150580:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4914.953244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4917.217020] LustreError: 149424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4917.246244] LustreError: 149424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4922.339157] LustreError: 147638:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002f19af39 x1874674920250240/t0(0) o250->MGC192.168.201.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4922.622177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4922.666859] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4924.256775] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4927.326827] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4927.982968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4927.993258] Lustre: Skipped 1 previous similar message [ 4928.026811] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4928.090369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4938.215163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4938.220722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4938.229270] Lustre: Skipped 1 previous similar message [ 4938.241611] Lustre: Skipped 1 previous similar message [ 4941.474289] Lustre: server umount lustre-OST0000 complete [ 4948.450326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4948.466315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4948.473050] Lustre: Skipped 1 previous similar message [ 4948.478421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4952.445801] Lustre: server umount lustre-MDT0000 complete [ 4953.580706] LustreError: 149425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4953.618769] LustreError: 149425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4956.546378] LustreError: 150221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829531 with bad export cookie 12151045616332454711 [ 4956.549063] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4956.554094] LustreError: 150221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4957.631152] Lustre: server umount lustre-MDT0001 complete [ 4969.584780] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 4973.998175] Key type lgssc unregistered [ 4974.533350] LNet: 155048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4974.558507] LNetError: 155048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4974.594846] LNet: Removed LNI 192.168.201.106@tcp [ 4975.906570] Key type .llcrypt unregistered [ 4975.908643] Key type ._llcrypt unregistered [ 5020.869509] Key type ._llcrypt registered [ 5020.872373] Key type .llcrypt registered [ 5021.079763] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 5023.032434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5023.158692] alg: No test for adler32 (adler32-zlib) [ 5024.287375] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 5024.494864] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 5026.138175] Key type lgssc registered [ 5027.998603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5037.246581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5037.299534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5038.998237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5044.769561] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5048.293785] LustreError: 156576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5048.311159] LustreError: 156576:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5053.409561] LustreError: 156577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5056.923182] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5058.528324] LustreError: 156576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5062.688317] LustreError: 156605:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5062.698336] LustreError: 156605:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5062.714676] LustreError: 156605:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 24, retries 0, failed: rc = -5 [ 5063.685919] Lustre: server umount lustre-MDT0000 complete [ 5080.469931] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 5091.716424] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 5131.481470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5131.789052] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5131.822362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5131.931347] Lustre: lustre-MDT0000: new disk, initializing [ 5132.005202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5132.020412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5136.576599] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5147.661524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5147.807888] Lustre: 159911:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5147.888023] LustreError: 159922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5147.913886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5147.921663] Lustre: Skipped 1 previous similar message [ 5148.019293] Lustre: lustre-MDT0001: new disk, initializing [ 5148.129765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5148.199529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5148.213269] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5153.087845] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5164.363170] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5170.845872] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5178.401612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5178.697525] Lustre: lustre-OST0000: new disk, initializing [ 5178.702942] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5178.710826] Lustre: 161879:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5178.796381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5184.455712] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5186.176930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5186.204835] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5186.289359] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5194.033660] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5198.869493] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5200.921647] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 07:22:53 (1787829773) [ 5452.773908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5452.775743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5452.788578] Lustre: Skipped 2 previous similar messages [ 5452.810092] Lustre: Skipped 2 previous similar messages [ 5457.895610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5457.907474] Lustre: Skipped 2 previous similar messages [ 5458.348618] Lustre: server umount lustre-MDT0000 complete [ 5462.743117] LustreError: 159904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830037 with bad export cookie 7351325904916793309 [ 5462.750791] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5462.756657] LustreError: 159904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5463.017268] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5463.018030] LustreError: 159918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5463.024810] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5467.040142] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5472.159739] LustreError: 159919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5472.159828] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5472.169739] LustreError: 159919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5477.344605] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5477.739903] Lustre: server umount lustre-MDT0001 complete [ 5487.880547] Lustre: server umount lustre-OST0000 complete [ 5498.367387] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 5509.616977] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 5549.419634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5549.797455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5549.905048] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5550.088763] Lustre: lustre-MDT0000: new disk, initializing [ 5550.195244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5550.246205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5554.814035] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5567.380466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5567.487456] Lustre: 166540:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5567.541799] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5567.549391] Lustre: Skipped 1 previous similar message [ 5567.619883] Lustre: lustre-MDT0001: new disk, initializing [ 5567.728423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5567.766658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5567.778817] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5573.729875] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5586.665727] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5594.151511] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5602.916578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5603.142740] Lustre: lustre-OST0000: new disk, initializing [ 5603.148920] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5603.156601] Lustre: 168509:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5603.254951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5610.622497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5610.637240] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5610.764946] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5611.389216] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5623.223539] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5629.064158] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5629.371876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5635.180417] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5635.439816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5641.701131] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5641.707906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5641.722138] LustreError: Skipped 1 previous similar message [ 5641.724954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5641.724967] Lustre: Skipped 1 previous similar message [ 5641.749819] Lustre: Skipped 1 previous similar message [ 5643.630630] Lustre: server umount lustre-OST0000 complete [ 5649.897621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5649.912744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5654.689953] Lustre: server umount lustre-MDT0000 complete [ 5657.056407] LustreError: 166548:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5657.088541] LustreError: 166548:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5659.108311] LustreError: 166532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830234 with bad export cookie 7351325904916952139 [ 5659.113702] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5659.122061] LustreError: 166532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5659.779803] Lustre: server umount lustre-MDT0001 complete [ 5669.015238] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:30:42 (1787830242) [ 5685.976436] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 5695.647146] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 5729.350294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5729.667980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5729.713967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5729.822857] Lustre: lustre-MDT0000: new disk, initializing [ 5729.940857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5729.970981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5735.691059] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5747.469660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5747.600883] Lustre: 173267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5747.911084] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5753.627781] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5763.694547] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5771.051878] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5778.496419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5778.806243] Lustre: lustre-OST0000: new disk, initializing [ 5778.809086] Lustre: Skipped 1 previous similar message [ 5778.815291] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5778.823108] Lustre: Skipped 2 previous similar messages [ 5778.831695] Lustre: 175247:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5786.295828] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5788.811485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5788.826405] Lustre: Skipped 1 previous similar message [ 5788.835550] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5788.947094] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5797.519139] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5809.635030] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5809.638567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5809.644144] Lustre: Skipped 2 previous similar messages [ 5809.648204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5809.653584] Lustre: Skipped 5 previous similar messages [ 5813.041215] Lustre: server umount lustre-OST0000 complete [ 5819.362788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5819.369414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5824.046522] Lustre: server umount lustre-MDT0000 complete [ 5824.995579] LustreError: 173279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5825.015835] LustreError: 173279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5828.013473] LustreError: 174068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830402 with bad export cookie 7351325904916953021 [ 5828.025373] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5828.038719] LustreError: 174068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5828.477620] Lustre: server umount lustre-MDT0001 complete [ 5838.529104] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:33:31 (1787830411) [ 5859.893621] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 5875.479917] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 5920.778128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5920.990390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5921.010653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5921.055269] Lustre: lustre-MDT0000: new disk, initializing [ 5921.116613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5921.121369] Lustre: Skipped 2 previous similar messages [ 5921.131809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5926.823510] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5938.157364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5938.270927] Lustre: 179593:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5938.434357] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5943.884735] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5954.807393] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5961.108792] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5968.301346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5968.592796] Lustre: 181570:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5975.254599] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5977.690418] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5977.777343] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5985.490039] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5998.560681] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5998.564500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5998.573194] LustreError: Skipped 1 previous similar message [ 5998.576897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5998.583504] Lustre: Skipped 2 previous similar messages [ 5998.615341] Lustre: Skipped 3 previous similar messages [ 5999.859474] Lustre: server umount lustre-OST0000 complete [ 6010.096499] Lustre: server umount lustre-MDT0000 complete [ 6013.897993] LustreError: 180394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830588 with bad export cookie 7351325904916954106 [ 6013.901449] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6013.907456] LustreError: 180394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6014.329723] Lustre: server umount lustre-MDT0001 complete [ 6027.728863] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 6030.596333] Key type lgssc unregistered [ 6030.875846] LNet: 183428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6030.887988] LNetError: 183428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6030.900068] LNet: Removed LNI 192.168.201.106@tcp [ 6031.864426] Key type .llcrypt unregistered [ 6031.866265] Key type ._llcrypt unregistered [ 6059.957318] Key type ._llcrypt registered [ 6059.959812] Key type .llcrypt registered [ 6060.051173] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 6061.479111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6061.542048] alg: No test for adler32 (adler32-zlib) [ 6062.819873] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 6063.137481] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 6064.807540] Key type lgssc registered [ 6066.091353] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6075.261096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6075.280753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6076.781131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6082.352832] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6086.118889] LustreError: 184846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6086.130915] LustreError: 184846:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6091.241887] LustreError: 184847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6093.512356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6093.753942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6098.602583] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6107.028150] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6112.902397] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6119.719778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6119.898598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6121.987193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6126.481499] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6134.512480] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6157.793382] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6157.810421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6157.820845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6158.821155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6158.824331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6158.839228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6163.814875] Lustre: server umount lustre-OST0000 complete [ 6169.070242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6169.091841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6169.104140] Lustre: Skipped 1 previous similar message [ 6169.124375] Lustre: Skipped 2 previous similar messages [ 6174.178853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6174.193410] Lustre: Skipped 1 previous similar message [ 6174.579638] Lustre: server umount lustre-MDT0000 complete [ 6178.274333] LustreError: 187285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6178.286456] LustreError: 187285:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6179.180504] LustreError: 186691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830754 with bad export cookie 7586255218137228329 [ 6179.199174] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6179.203019] LustreError: 186691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6179.566695] Lustre: server umount lustre-MDT0001 complete [ 6193.038709] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 6195.905377] Key type lgssc unregistered [ 6196.224880] LNet: 188409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6196.233056] LNetError: 188409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6196.245243] LNet: Removed LNI 192.168.201.106@tcp [ 6197.021280] Key type .llcrypt unregistered [ 6197.022636] Key type ._llcrypt unregistered [ 6228.025316] Key type ._llcrypt registered [ 6228.029542] Key type .llcrypt registered [ 6228.172648] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 6229.503454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6229.580715] alg: No test for adler32 (adler32-zlib) [ 6230.630276] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 6230.881189] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 6232.569083] Key type lgssc registered [ 6233.551321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6241.669335] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6241.684615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6243.086669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6247.477173] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6252.521670] LustreError: 189827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6252.553882] LustreError: 189827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6256.646889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6257.102755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6262.425650] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6271.605586] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6278.853990] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6288.434983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6288.900611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6295.642533] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6304.167281] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6312.434328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6322.671574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6322.678776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6322.694527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6322.710685] Lustre: Skipped 1 previous similar message [ 6326.411245] Lustre: lustre-OST0000: Not available for connect from 192.168.201.6@tcp (stopping) [ 6327.783968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6327.794891] Lustre: Skipped 1 previous similar message [ 6327.956406] Lustre: server umount lustre-OST0000 complete [ 6336.224929] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8f7711690800.ost_server_uuid 50 [ 6337.942711] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6354.400765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6354.413824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6354.428962] Lustre: Skipped 1 previous similar message [ 6354.433335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6357.472500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6360.154881] Lustre: server umount lustre-MDT0000 complete [ 6362.593067] LustreError: 190792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6362.625421] LustreError: 190792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6364.106444] LustreError: 189807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830939 with bad export cookie 8006754935216186675 [ 6364.108300] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6364.117937] LustreError: 189807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6364.611935] Lustre: server umount lustre-MDT0001 complete [ 6374.912063] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 07:42:28 (1787830948) [ 6382.071953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6382.483982] LustreError: 193770:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6382.565773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6387.301628] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6387.683040] LustreError: 193771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6392.805328] LustreError: 193770:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6396.637409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6397.127286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6401.886741] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6409.919398] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6416.455068] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6425.162031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6425.737515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6427.855652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6432.372702] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6440.436051] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6447.852386] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6448.125378] Lustre: lustre-OST0001: new disk, initializing [ 6448.128557] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6448.133255] Lustre: 196611:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6448.225620] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6454.901245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6454.918592] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6454.948843] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6455.035714] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6465.369212] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6469.089700] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6469.097064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6469.106238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6469.111448] Lustre: Skipped 2 previous similar messages [ 6474.631065] Lustre: server umount lustre-OST0000 complete [ 6475.766894] LustreError: 196268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6475.784280] LustreError: 196268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6482.267127] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6482.460366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6487.444879] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6487.775682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6496.227052] LustreError: 196268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6496.256983] LustreError: 196268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6506.465233] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6506.474619] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6506.480891] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6506.485194] Lustre: Skipped 2 previous similar messages [ 6506.504212] Lustre: Skipped 4 previous similar messages [ 6509.480785] Lustre: server umount lustre-OST0001 complete [ 6515.182680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6515.200633] LustreError: Skipped 1 previous similar message [ 6515.209344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6519.980427] Lustre: server umount lustre-MDT0000 complete [ 6524.685340] LustreError: 193750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831099 with bad export cookie 8006754935216187746 [ 6524.687415] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6524.698738] LustreError: 193750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6524.997956] Lustre: server umount lustre-MDT0001 complete [ 6550.817201] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 07:45:23 (1787831123) [ 6557.240846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6557.643821] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6557.679878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6557.839544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6563.089562] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6572.000423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6572.087936] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6572.137161] Lustre: 200047:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6576.592672] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6584.262308] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6590.136557] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6596.966605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6597.126474] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6602.406506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6603.706125] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6611.227391] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6618.452734] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6618.554713] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6626.142359] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6634.898317] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6640.121875] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6640.137539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6640.154700] Lustre: Skipped 2 previous similar messages [ 6640.170392] Lustre: Skipped 4 previous similar messages [ 6644.784325] Lustre: server umount lustre-OST0000 complete [ 6645.221565] LustreError: 203597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6645.249662] LustreError: 203597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6665.702208] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6665.713414] LustreError: Skipped 1 previous similar message [ 6670.331773] Lustre: server umount lustre-OST0001 complete [ 6675.937753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6675.940045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6675.955567] Lustre: Skipped 3 previous similar messages [ 6675.978477] Lustre: Skipped 4 previous similar messages [ 6681.441663] Lustre: server umount lustre-MDT0000 complete [ 6685.330716] LustreError: 201918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831260 with bad export cookie 8006754935216189153 [ 6685.335973] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6685.342420] LustreError: 201918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6685.699759] Lustre: server umount lustre-MDT0001 complete [ 6710.672548] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 07:48:03 (1787831283) [ 6726.892596] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 6739.806286] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 6778.022281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6778.221862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6778.258452] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6778.339514] Lustre: lustre-MDT0000: new disk, initializing [ 6778.454242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6778.457353] Lustre: Skipped 3 previous similar messages [ 6778.490559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6783.549746] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6795.366142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6795.481128] Lustre: 208146:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6795.554283] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6795.564086] Lustre: Skipped 1 previous similar message [ 6795.658259] Lustre: lustre-MDT0001: new disk, initializing [ 6795.797871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6795.828575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6800.653424] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6809.619177] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6817.105774] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6825.249370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6825.714619] Lustre: lustre-OST0000: new disk, initializing [ 6825.717823] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6825.722812] Lustre: 210117:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6832.212715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6832.219857] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6832.329927] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6833.111903] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6844.791580] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6850.651328] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6850.845279] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6855.276169] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6855.505236] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6858.210583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6858.212190] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6858.240769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6858.240778] Lustre: Skipped 2 previous similar messages [ 6863.734652] Lustre: server umount lustre-OST0000 complete [ 6874.023989] Lustre: server umount lustre-MDT0000 complete [ 6878.191435] LustreError: 208140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831453 with bad export cookie 8006754935216190714 [ 6878.196380] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6878.204256] LustreError: 208140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6878.648321] Lustre: server umount lustre-MDT0001 complete [ 6888.152957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6888.787049] LustreError: 212335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6888.822057] LustreError: 212335:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6894.767608] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6904.913867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6909.837393] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6918.012379] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6924.834216] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6932.494298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6932.980532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6932.983422] Lustre: Skipped 4 previous similar messages [ 6939.871666] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6948.208152] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6953.588216] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6953.796477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6958.457739] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6958.795057] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6964.706749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6967.848842] Lustre: server umount lustre-OST0000 complete [ 6975.829474] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6980.533348] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7012.260715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7015.865590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7015.889975] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7020.280265] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7028.339817] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7029.774388] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7030.261770] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7030.294480] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7036.387340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7036.393837] LustreError: Skipped 1 previous similar message [ 7036.395635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7036.416185] Lustre: Skipped 5 previous similar messages [ 7036.421224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7036.427951] Lustre: Skipped 10 previous similar messages [ 7041.611635] Lustre: server umount lustre-OST0000 complete [ 7051.991871] Lustre: server umount lustre-MDT0000 complete [ 7055.841400] LustreError: 213063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7055.849570] LustreError: 213063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7056.490373] LustreError: 212315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831631 with bad export cookie 8006754935216191596 [ 7056.494760] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7056.503903] LustreError: 212315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7056.932195] Lustre: server umount lustre-MDT0001 complete [ 7067.283054] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 07:54:00 (1787831640) [ 7075.115362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7079.765568] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7087.910422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7092.899332] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7101.415199] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7108.155858] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7115.364653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7121.750412] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7130.445974] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7135.108879] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7135.341395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7139.795957] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7139.982828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7146.756120] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7146.949509] Lustre: lustre-OST0001: new disk, initializing [ 7146.953184] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7146.958146] Lustre: 221673:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7151.941656] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7153.225811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7153.236228] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7153.310138] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7160.541594] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7164.639620] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7164.809553] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7168.752706] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7168.970202] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7174.115924] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7174.138286] LustreError: Skipped 2 previous similar messages [ 7185.375132] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7185.658445] Lustre: server umount lustre-OST0001 complete [ 7193.757823] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7194.064316] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7200.275909] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7200.451718] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7231.666931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7231.993154] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7232.008776] Lustre: Skipped 5 previous similar messages [ 7237.109449] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7237.137053] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7237.137441] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7237.172489] Lustre: Skipped 1 previous similar message [ 7239.802282] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7249.060905] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7251.364963] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7259.335312] Lustre: server umount lustre-OST0001 complete [ 7291.360857] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7291.615255] Lustre: server umount lustre-OST0000 complete [ 7298.016593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7298.022536] Lustre: Skipped 9 previous similar messages [ 7298.028915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7298.040656] Lustre: Skipped 20 previous similar messages [ 7310.303420] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7310.519871] Lustre: server umount lustre-MDT0000 complete [ 7313.889340] LustreError: 218304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7313.896499] LustreError: 218304:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 40 previous similar messages [ 7314.106735] LustreError: 218286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831889 with bad export cookie 8006754935216192562 [ 7314.108133] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7314.116296] LustreError: 218286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7320.557664] Lustre: server umount lustre-MDT0001 complete [ 7346.955546] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 07:58:39 (1787831919) [ 7354.952665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7355.321061] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7355.387553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7360.725900] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7368.995526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7369.067154] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7369.128323] Lustre: 226923:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7373.698653] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7382.098642] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7388.364476] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7396.829941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7397.107139] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7403.933912] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7411.937381] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7421.978333] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7422.078804] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7429.756975] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7439.124639] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7441.683734] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7447.204953] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7447.528117] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7453.062418] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7453.266692] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7460.157732] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f7708ad3000.ost_server_uuid 50 [ 7461.763329] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f7708ad3000.ost_server_uuid in IDLE state after 0 sec [ 7463.050830] Lustre: Permanently deactivating lustre-OST0001 [ 7463.084598] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7467.265535] Lustre: Permanently reactivating lustre-OST0001 [ 7467.309972] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7467.319164] Lustre: Skipped 2 previous similar messages [ 7472.637117] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7472.650819] LustreError: Skipped 5 previous similar messages [ 7477.980612] Lustre: server umount lustre-OST0001 complete [ 7488.564361] Lustre: server umount lustre-OST0000 complete [ 7498.892685] Lustre: server umount lustre-MDT0000 complete [ 7503.830187] LustreError: 227665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832078 with bad export cookie 8006754935216193850 [ 7503.836114] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7503.842364] LustreError: 227665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7504.159696] Lustre: server umount lustre-MDT0001 complete [ 7532.130091] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 08:01:45 (1787832105) [ 7541.302451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7541.797477] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7541.865065] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7541.880746] Lustre: Skipped 2 previous similar messages [ 7546.710646] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7555.689382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7555.907752] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7555.960597] Lustre: 233631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7562.566898] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7573.284919] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7579.637586] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7588.149201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7588.452600] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7588.509158] Lustre: Permanently deactivating lustre-OST0000 [ 7588.546976] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7595.224539] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7595.566726] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7603.696837] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7611.458986] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7611.514238] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7617.528838] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7620.801771] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7625.470367] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7627.801799] Lustre: Permanently reactivating lustre-OST0000 [ 7627.820909] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7627.827765] Lustre: Skipped 2 previous similar messages [ 7628.768932] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7628.788803] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 33 seconds. I think it's dead, and I am evicting it. exp 00000000f5fa0554, cur 1787832203 deadline 1787832200 last 1787832170 [ 7628.878769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7647.588232] Lustre: server umount lustre-OST0001 complete [ 7657.805599] Lustre: server umount lustre-OST0000 complete [ 7676.383781] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7676.591776] Lustre: server umount lustre-MDT0000 complete [ 7681.343857] LustreError: 233624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832256 with bad export cookie 8006754935216195523 [ 7681.353892] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7682.265880] Lustre: server umount lustre-MDT0001 complete [ 7700.103555] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 7703.290741] Key type lgssc unregistered [ 7703.714955] LNet: 238541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7703.719635] LNetError: 238541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7703.736022] LNet: Removed LNI 192.168.201.106@tcp [ 7704.488143] Key type .llcrypt unregistered [ 7704.490676] Key type ._llcrypt unregistered [ 7729.698781] Key type ._llcrypt registered [ 7729.704231] Key type .llcrypt registered [ 7729.822407] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 7731.131549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7731.161176] alg: No test for adler32 (adler32-zlib) [ 7732.269351] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 7732.523343] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 7734.208989] Key type lgssc registered [ 7735.467351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7746.455903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7746.470852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7747.946286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7754.510540] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7757.283545] LustreError: 240012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7757.312161] LustreError: 240012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7762.413368] LustreError: 240011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7764.740608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7764.876643] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7764.902979] Lustre: 239999:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7764.922682] Lustre: Permanently deactivating lustre-MDT0001 [ 7764.938442] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7765.263522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7766.095617] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7766.108214] LustreError: 240042:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7766.130270] LustreError: 240042:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7766.145801] LustreError: 240042:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 1, failed: rc = -108 [ 7770.198162] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7779.108477] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7785.958712] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7794.304467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7794.837082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7801.744901] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7809.587382] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7810.147146] LustreError: 241853:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7810.164107] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7815.655231] LustreError: 241853:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7817.584042] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7817.720539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7817.801909] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7822.869338] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7822.875017] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7825.788165] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7835.102062] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7838.823947] Lustre: Permanently reactivating lustre-MDT0001 [ 7838.862226] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7838.872279] Lustre: Skipped 2 previous similar messages [ 7845.519649] Lustre: lustre-MDT0001: Client 79b54380-2030-45cd-98e5-05c9853204b5 (at 192.168.201.6@tcp) reconnecting [ 7852.484334] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7854.201378] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7860.284410] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7860.497869] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7865.118227] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7865.315168] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7867.889936] Lustre: Permanently deactivating lustre-MDT0001 [ 7867.906268] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7867.911456] Lustre: Skipped 2 previous similar messages [ 7877.265102] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7883.235058] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7883.239920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7883.253594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7884.266371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7884.268872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7884.285976] Lustre: Skipped 1 previous similar message [ 7884.393649] Lustre: server umount lustre-MDT0000 complete [ 7888.090546] LustreError: 243560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832462 with bad export cookie 2884408745725538930 [ 7888.099890] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7888.106517] LustreError: 243560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7888.403753] Lustre: server umount lustre-MDT0001 complete [ 7894.381505] Lustre: server umount lustre-OST0000 complete [ 7908.489655] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 08:08:00 (1787832480) [ 7920.907095] Lustre: server umount lustre-OST0001 complete [ 7929.703856] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 7943.018815] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 7992.199176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7992.595197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7992.606785] Lustre: Skipped 2 previous similar messages [ 7992.643459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7992.785950] Lustre: lustre-MDT0000: new disk, initializing [ 7992.922358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7992.971890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8000.448429] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8012.378673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8012.666897] Lustre: 248271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8012.796597] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8012.805570] Lustre: Skipped 1 previous similar message [ 8012.941194] Lustre: lustre-MDT0001: new disk, initializing [ 8013.086883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8013.154951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8013.164479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8018.450557] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8028.864903] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8034.512929] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8043.145726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8043.582501] Lustre: lustre-OST0000: new disk, initializing [ 8043.590028] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8043.599113] Lustre: 250243:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8043.737315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8052.362503] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8052.816297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8052.821871] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8053.008739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8065.362551] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8067.966805] LustreError: 248279:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8077.979725] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8078.058690] LustreError: 248279:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8078.091462] LustreError: 248281:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8078.271082] Lustre: lustre-OST0001: new disk, initializing [ 8078.275743] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8078.285937] Lustre: 251359:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8078.424483] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8084.706914] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8087.183360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8087.200249] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8087.305530] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8088.182576] LustreError: 248281:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8095.039106] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8097.487892] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8102.880509] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8102.891808] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8102.897338] LustreError: Skipped 1 previous similar message [ 8102.907441] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8102.916758] Lustre: Skipped 3 previous similar messages [ 8102.944725] Lustre: Skipped 1 previous similar message [ 8108.016127] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8108.020659] Lustre: Skipped 2 previous similar messages [ 8113.166399] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8113.188465] Lustre: Skipped 2 previous similar messages [ 8114.145031] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8114.293968] LustreError: 250229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8114.298110] Lustre: server umount lustre-OST0001 complete [ 8123.365324] LustreError: 251820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8123.396596] LustreError: 251820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8133.603628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8133.627667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8140.103964] Lustre: server umount lustre-OST0000 complete [ 8146.400453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8146.406076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8146.432104] Lustre: Skipped 1 previous similar message [ 8154.083140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8154.096435] Lustre: Skipped 7 previous similar messages [ 8160.223562] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8160.591434] Lustre: server umount lustre-MDT0000 complete [ 8163.303515] LustreError: 248281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8163.327187] LustreError: 248281:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 8165.695408] LustreError: 248264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832740 with bad export cookie 2884408745725541744 [ 8165.711383] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8165.712608] LustreError: 248264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8172.389240] Lustre: server umount lustre-MDT0001 complete [ 8186.520784] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 8189.310047] Key type lgssc unregistered [ 8189.610072] LNet: 253399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8189.628634] LNetError: 253399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8189.644836] LNet: Removed LNI 192.168.201.106@tcp [ 8190.485408] Key type .llcrypt unregistered [ 8190.487026] Key type ._llcrypt unregistered [ 8230.838752] Key type ._llcrypt registered [ 8230.840770] Key type .llcrypt registered [ 8231.003354] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8232.387988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8232.441533] alg: No test for adler32 (adler32-zlib) [ 8233.566399] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 8233.793664] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 8235.463527] Key type lgssc registered [ 8236.530838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8245.283738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8245.299910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8246.562957] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8246.590485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8246.778505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8251.458695] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8260.500527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8260.562256] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8260.590988] Lustre: 255611:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8260.900966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8265.836853] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8274.235877] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8280.301822] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8289.497439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8289.833814] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8290.145434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8297.115342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8297.404936] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8305.188751] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8358.373894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8358.389429] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8358.407832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8364.001692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8364.013620] Lustre: Skipped 2 previous similar messages [ 8364.504590] Lustre: server umount lustre-OST0000 complete [ 8369.594146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8386.163116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8386.646952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8389.671828] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8389.692675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8389.705361] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8389.725399] Lustre: Skipped 1 previous similar message [ 8393.182696] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8402.040964] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8445.920578] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8445.923660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8445.928606] LustreError: Skipped 1 previous similar message [ 8445.932117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8445.958282] Lustre: Skipped 2 previous similar messages [ 8445.981717] Lustre: Skipped 1 previous similar message [ 8449.744714] Lustre: server umount lustre-OST0000 complete [ 8455.648303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8455.654761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8455.665568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8460.156734] Lustre: server umount lustre-MDT0000 complete [ 8464.241612] LustreError: 255602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833039 with bad export cookie 18030250962327144266 [ 8464.252884] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8464.258777] LustreError: 255602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8464.815299] Lustre: server umount lustre-MDT0001 complete [ 8477.390946] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 8480.617674] Key type lgssc unregistered [ 8480.997569] LNet: 261038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8481.027740] LNetError: 261038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8482.093917] LNet: Removed LNI 192.168.201.106@tcp [ 8483.228915] Key type .llcrypt unregistered [ 8483.232787] Key type ._llcrypt unregistered [ 8512.063170] Key type ._llcrypt registered [ 8512.069145] Key type .llcrypt registered [ 8512.335997] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8514.385120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8514.614347] alg: No test for adler32 (adler32-zlib) [ 8515.769397] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 8516.089480] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 8517.759177] Key type lgssc registered [ 8518.979640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8526.522989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8526.550543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8528.230918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8533.863396] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8538.604994] LustreError: 262455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8538.627151] LustreError: 262455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8543.715135] LustreError: 262456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8545.741879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8546.473906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8553.159609] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8561.698608] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8569.426815] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8578.333195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8578.969483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8588.224222] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8598.531038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8599.182382] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8628.193880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8628.202971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8628.213887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8629.234333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8629.235875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8629.251765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8629.282101] Lustre: Skipped 1 previous similar message [ 8632.294888] Lustre: server umount lustre-OST0000 complete [ 8638.433037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8638.440690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8638.456314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8642.715539] Lustre: server umount lustre-MDT0000 complete [ 8644.576986] LustreError: 262456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8644.595627] LustreError: 262456:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8647.052321] LustreError: 262435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833221 with bad export cookie 1435782971559795861 [ 8647.064243] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8647.532361] Lustre: server umount lustre-MDT0001 complete [ 8659.793207] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 8662.562963] Key type lgssc unregistered [ 8662.838898] LNet: 266769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8662.841695] LNetError: 266769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8662.860660] LNet: Removed LNI 192.168.201.106@tcp [ 8663.964430] Key type .llcrypt unregistered [ 8663.968659] Key type ._llcrypt unregistered [ 8683.249642] Key type ._llcrypt registered [ 8683.252302] Key type .llcrypt registered [ 8683.343871] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8684.314320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8684.372535] alg: No test for adler32 (adler32-zlib) [ 8685.572478] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 8685.790674] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 8687.455657] Key type lgssc registered [ 8688.739421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8697.086164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8697.132752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8698.723551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8703.447549] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8709.098318] LustreError: 267992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8709.118399] LustreError: 267992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8713.330740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8713.829719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8720.156201] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8728.336246] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8735.739930] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8743.791329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8744.240178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8745.264428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8751.033362] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8758.780307] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8771.040545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8771.059094] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8771.097729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8772.866388] Lustre: server umount lustre-OST0000 complete [ 8779.262106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8779.272273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8779.274351] Lustre: Skipped 1 previous similar message [ 8779.298061] Lustre: Skipped 1 previous similar message [ 8784.196741] Lustre: server umount lustre-MDT0000 complete [ 8784.358791] LustreError: 267992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8784.387832] LustreError: 267992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8789.473541] LustreError: 270113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8789.504074] LustreError: 270113:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8790.053292] LustreError: 267972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833364 with bad export cookie 4958142293720081294 [ 8790.067083] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8790.071091] LustreError: 267972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8799.427324] Lustre: server umount lustre-MDT0001 complete [ 8810.974770] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 8813.646131] Key type lgssc unregistered [ 8813.978428] LNet: 271676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8813.986915] LNetError: 271676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8814.013382] LNet: Removed LNI 192.168.201.106@tcp [ 8814.759433] Key type .llcrypt unregistered [ 8814.761598] Key type ._llcrypt unregistered [ 8840.157932] Key type ._llcrypt registered [ 8840.161167] Key type .llcrypt registered [ 8840.273349] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8841.499225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8841.530534] alg: No test for adler32 (adler32-zlib) [ 8842.734559] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 8842.968036] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 8844.607341] Key type lgssc registered [ 8846.131298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8854.293051] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8854.320457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8855.883283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8860.826579] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8865.249903] LustreError: 273095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8865.270768] LustreError: 273095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8869.842949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8870.224679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8873.996524] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8882.541772] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8889.641723] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8898.144054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8898.733503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8900.827791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8906.037800] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8915.698318] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8942.051697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8942.076983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8942.104241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8946.148772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8946.163028] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8946.174170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8947.820745] Lustre: server umount lustre-OST0000 complete [ 8952.288506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8952.297643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8952.318857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8952.326134] Lustre: Skipped 1 previous similar message [ 8956.385683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8956.388486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8956.407182] Lustre: Skipped 1 previous similar message [ 8957.865819] Lustre: server umount lustre-MDT0000 complete [ 8961.506636] LustreError: 274149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8961.533879] LustreError: 274149:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8962.119104] LustreError: 273076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833537 with bad export cookie 10024757298686639886 [ 8962.126813] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8962.131590] LustreError: 273076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8962.467706] Lustre: server umount lustre-MDT0001 complete [ 8973.631182] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 8976.510225] Key type lgssc unregistered [ 8976.764831] LNet: 277406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8976.771730] LNetError: 277406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8976.786494] LNet: Removed LNI 192.168.201.106@tcp [ 8977.742609] Key type .llcrypt unregistered [ 8977.745707] Key type ._llcrypt unregistered [ 8997.352212] Key type ._llcrypt registered [ 8997.353838] Key type .llcrypt registered [ 8997.472542] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8998.471841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8998.534639] alg: No test for adler32 (adler32-zlib) [ 8999.734653] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 9000.033772] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 9001.735141] Key type lgssc registered [ 9002.885875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9012.363528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9012.374473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9014.049708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9018.985380] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9023.458817] LustreError: 278643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9023.483019] LustreError: 278643:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9028.586739] LustreError: 278645:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9029.058308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9029.442106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9034.112379] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9042.451524] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9048.351126] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9053.784936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9054.061769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9058.172518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9058.987947] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9064.849491] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9075.685222] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9075.694384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9075.728676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9078.759646] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9078.765383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9078.781790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9081.671784] Lustre: server umount lustre-OST0000 complete [ 9085.920179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9085.929575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9085.970219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9085.978904] Lustre: Skipped 1 previous similar message [ 9088.993943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9088.998994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9089.015948] Lustre: Skipped 1 previous similar message [ 9091.663361] Lustre: server umount lustre-MDT0000 complete [ 9094.116239] LustreError: 278630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9094.145593] LustreError: 278630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9095.670970] LustreError: 278610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833670 with bad export cookie 8762115348916082468 [ 9095.675861] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9095.684343] LustreError: 278610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9096.124255] Lustre: server umount lustre-MDT0001 complete [ 9108.030669] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 9110.526566] Key type lgssc unregistered [ 9110.765679] LNet: 282347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9110.769119] LNetError: 282347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9110.780263] LNet: Removed LNI 192.168.201.106@tcp [ 9111.549390] Key type .llcrypt unregistered [ 9111.555472] Key type ._llcrypt unregistered [ 9124.452711] Key type ._llcrypt registered [ 9124.457811] Key type .llcrypt registered [ 9124.625771] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9144.777512] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 9160.116502] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9161.476242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9161.573326] alg: No test for adler32 (adler32-zlib) [ 9162.684581] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 9162.893715] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 9164.551204] Key type lgssc registered [ 9165.860371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9206.124715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9206.139986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9207.413840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9207.464072] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9207.594428] Lustre: lustre-MDT0000: new disk, initializing [ 9207.714122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9207.742409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9212.006983] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9222.047602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9222.186258] Lustre: 285901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9222.240876] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9222.244971] Lustre: Skipped 1 previous similar message [ 9222.333890] Lustre: lustre-MDT0001: new disk, initializing [ 9222.419535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9222.462771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9222.478631] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9226.977688] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9237.106832] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9243.204410] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9249.972146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9250.229519] Lustre: lustre-OST0000: new disk, initializing [ 9250.236262] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9250.243726] Lustre: 287868:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9250.346430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9256.198151] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9259.082658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9259.091493] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9259.164719] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9265.213176] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9269.830545] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9270.072616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9274.485644] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9274.647845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9279.983183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9279.983216] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9279.994529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9280.003934] Lustre: Skipped 1 previous similar message [ 9282.681478] Lustre: server umount lustre-OST0000 complete [ 9289.188353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9289.194720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9289.216381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9289.223063] Lustre: Skipped 1 previous similar message [ 9290.213041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9290.227596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9290.231972] Lustre: Skipped 1 previous similar message [ 9293.124575] Lustre: server umount lustre-MDT0000 complete [ 9296.818040] LustreError: 285894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833871 with bad export cookie 8432818537545451300 [ 9296.833817] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9296.839907] LustreError: 285894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9297.445433] Lustre: server umount lustre-MDT0001 complete [ 9307.105550] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 08:31:20 (1787833880) [ 9314.364978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9314.909553] LustreError: 290279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9314.933824] LustreError: 290279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9315.002565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9319.362644] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9320.427956] LustreError: 290280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9324.526110] LustreError: 290279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9327.019981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9327.406102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9332.684583] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9340.889377] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9346.139646] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9354.122504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9354.859792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9362.300384] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9371.064259] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9382.884783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9382.889579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9382.909409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9387.906926] Lustre: server umount lustre-OST0000 complete [ 9393.122653] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9393.141517] Lustre: Skipped 1 previous similar message [ 9393.154486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9393.166438] Lustre: Skipped 2 previous similar messages [ 9399.195409] Lustre: server umount lustre-MDT0000 complete [ 9402.338245] LustreError: 290280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9402.358667] LustreError: 290280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9403.817236] LustreError: 290260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833978 with bad export cookie 8432818537545452189 [ 9403.822727] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9403.826179] LustreError: 290260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9404.073639] Lustre: server umount lustre-MDT0001 complete [ 9416.299522] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 9419.068417] Key type lgssc unregistered [ 9419.387823] LNet: 293820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9419.393119] LNetError: 293820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9419.424854] LNet: Removed LNI 192.168.201.106@tcp [ 9420.364543] Key type .llcrypt unregistered [ 9420.367077] Key type ._llcrypt unregistered [ 9446.530703] Key type ._llcrypt registered [ 9446.533969] Key type .llcrypt registered [ 9446.644973] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 9461.478858] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9462.373995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9462.416239] alg: No test for adler32 (adler32-zlib) [ 9463.541803] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 9463.846643] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 9465.551173] Key type lgssc registered [ 9466.612643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9515.505597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9515.534197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9516.820720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9516.845499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9516.915487] Lustre: lustre-MDT0000: new disk, initializing [ 9516.969150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9516.982280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9521.184973] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9531.765048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9531.887864] Lustre: 297650:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9531.940691] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9531.944716] Lustre: Skipped 1 previous similar message [ 9532.028713] Lustre: lustre-MDT0001: new disk, initializing [ 9532.148144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9532.181897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9532.196906] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9537.916487] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9547.468478] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9553.672441] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9560.447885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9560.686341] Lustre: lustre-OST2710: new disk, initializing [ 9560.689981] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9560.695542] Lustre: 299617:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9560.805931] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9567.331059] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9567.781728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9567.788241] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9567.934063] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9577.181386] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9583.447944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9583.547961] Lustre: lustre-OST03e8: new disk, initializing [ 9583.550349] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9583.553858] Lustre: 300681:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9583.629854] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9589.646180] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9591.876776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9591.888989] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9591.966512] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9597.501932] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9605.383776] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9605.567947] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9609.996296] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9610.252126] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9615.834977] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9615.987194] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9621.525437] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9621.780633] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9628.138749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9628.160233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9628.171197] Lustre: Skipped 2 previous similar messages [ 9628.190751] Lustre: Skipped 3 previous similar messages [ 9633.254716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9636.052319] Lustre: server umount lustre-MDT0000 complete [ 9640.183731] LustreError: 297641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834215 with bad export cookie 16510755118981799885 [ 9640.192601] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9640.194707] LustreError: 297641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9640.538827] Lustre: server umount lustre-MDT0001 complete [ 9655.227510] Lustre: server umount lustre-OST2710 complete [ 9659.359522] Lustre: 295446:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787834218/real 1787834218] req@0000000014db9691 x1874679864300800/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787834234 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9659.375492] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9659.382119] Lustre: Skipped 1 previous similar message [ 9659.715919] Lustre: server umount lustre-OST03e8 complete [ 9682.837187] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 9694.749713] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9736.051244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9736.557758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9736.617643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9736.723968] Lustre: lustre-MDT0000: new disk, initializing [ 9736.833695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9736.891036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9741.842762] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9753.644672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9753.774384] Lustre: 305933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9753.824533] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9753.827894] Lustre: Skipped 1 previous similar message [ 9753.892334] Lustre: lustre-MDT0001: new disk, initializing [ 9753.982414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9754.016544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9754.036095] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9758.537915] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9770.469688] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9776.893276] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9784.665206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9785.052321] Lustre: lustre-OST0000: new disk, initializing [ 9785.056518] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9785.064185] Lustre: 307905:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9785.151596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9792.661720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9792.674827] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9792.801876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9793.253248] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9803.493589] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9809.269313] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9809.481613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9816.382595] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9816.736812] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9823.718251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9823.721919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9823.731553] LustreError: Skipped 1 previous similar message [ 9823.757545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9823.767290] Lustre: Skipped 3 previous similar messages [ 9825.611915] Lustre: server umount lustre-OST0000 complete [ 9830.880356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9830.887438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9830.897600] Lustre: Skipped 1 previous similar message [ 9830.902428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9830.906485] Lustre: Skipped 1 previous similar message [ 9835.741409] Lustre: server umount lustre-MDT0000 complete [ 9838.977385] LustreError: 306729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834413 with bad export cookie 16510755118981801579 [ 9838.981289] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9838.989977] LustreError: 306729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9839.337918] Lustre: server umount lustre-MDT0001 complete [ 9848.103963] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 08:40:21 (1787834421) [ 9849.573953] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9851.410296] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 08:40:24 (1787834424) [ 9854.809426] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9881.315836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9881.537995] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9881.554880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9881.774541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9886.742558] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9895.392783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9895.534309] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9895.591321] Lustre: 311763:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9900.474720] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9922.673058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9922.952446] Lustre: Denying initial registration attempt from nid 192.168.201.106@tcp, specified as failover [ 9922.961124] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9922.977977] LustreError: 313216:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9922.993538] LustreError: 313216:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9923.000262] LustreError: 313216:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9923.080329] Lustre: server umount lustre-OST0000 complete [ 9923.082531] LustreError: 313216:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9931.747632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9931.754736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9931.770657] Lustre: Skipped 1 previous similar message [ 9931.777378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9931.779679] Lustre: Skipped 2 previous similar messages [ 9936.418745] Lustre: server umount lustre-MDT0000 complete [ 9939.426229] LustreError: 311774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9939.445408] LustreError: 311774:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9939.934205] LustreError: 311756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834514 with bad export cookie 16510755118981802461 [ 9939.941053] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9939.947785] LustreError: 311756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9940.271878] Lustre: server umount lustre-MDT0001 complete [ 9951.175905] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 9953.595956] Key type lgssc unregistered [ 9953.905596] LNet: 314340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9953.913204] LNetError: 314340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9953.926990] LNet: Removed LNI 192.168.201.106@tcp [ 9954.738114] Key type .llcrypt unregistered [ 9954.739637] Key type ._llcrypt unregistered [ 9968.072137] Key type ._llcrypt registered [ 9968.073687] Key type .llcrypt registered [ 9968.268785] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9970.436404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9970.496049] alg: No test for adler32 (adler32-zlib) [ 9971.664669] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [ 9971.958678] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 9973.679657] Key type lgssc registered [ 9975.288854] Lustre: Echo OBD driver; http://www.lustre.org/ [10024.922356] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [10035.659209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10035.698056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10037.023657] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10037.063063] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10037.209072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10041.634276] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10051.897631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10051.994462] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10052.025298] Lustre: 317102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10052.278858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10057.775261] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10065.996924] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10071.953387] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10092.072792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10092.296906] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10092.483872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10099.091468] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10107.421753] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10109.921363] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10109.928821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10109.956772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10110.455904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10110.461904] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10110.474494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10110.479563] Lustre: Skipped 1 previous similar message [10115.629084] Lustre: server umount lustre-OST0000 complete [10120.672752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10120.690649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10120.707146] Lustre: Skipped 1 previous similar message [10125.793834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10125.809880] Lustre: Skipped 1 previous similar message [10126.313044] Lustre: server umount lustre-MDT0000 complete [10130.735420] LustreError: 317094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834705 with bad export cookie 17578963997062501735 [10130.746609] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10130.747376] LustreError: 317094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10131.065386] Lustre: server umount lustre-MDT0001 complete [10143.589421] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [10146.524445] Key type lgssc unregistered [10146.821801] LNet: 320679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10146.829907] LNetError: 320679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10146.857816] LNet: Removed LNI 192.168.201.106@tcp [10147.955709] Key type .llcrypt unregistered [10147.958049] Key type ._llcrypt unregistered [10175.501972] Key type ._llcrypt registered [10175.504399] Key type .llcrypt registered [10175.649579] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [10176.779731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10176.903056] alg: No test for adler32 (adler32-zlib) [10178.018243] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [10178.335727] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [10180.023168] Key type lgssc registered [10181.050652] Lustre: Echo OBD driver; http://www.lustre.org/ [10188.310680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10188.335921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10189.663583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10194.169456] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10200.039109] LustreError: 322096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10200.053134] LustreError: 322096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10203.738227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10204.127811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10208.849563] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10217.893747] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10224.272543] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10233.003137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10233.442711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10239.859275] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10247.192717] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10259.427445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10259.429540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10259.442087] Lustre: Skipped 2 previous similar messages [10259.459816] Lustre: Skipped 2 previous similar messages [10260.640645] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.6@tcp (stopping) [10262.563461] Lustre: server umount lustre-MDT0000 complete [10264.556024] LustreError: 322093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10264.589609] LustreError: 322093:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10265.707480] LustreError: 324539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10266.915547] LustreError: 322078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834841 with bad export cookie 13827720234696277332 [10266.923791] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10269.665883] LustreError: 322092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10269.668643] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10269.699622] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10271.721311] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10271.724047] Lustre: Skipped 1 previous similar message [10273.496523] Lustre: server umount lustre-MDT0001 complete [10281.842820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10291.543442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10292.130070] LustreError: 321408:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000095be088a x1874680614002944/t0(0) o250->MGC192.168.201.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10292.740246] LustreError: 323928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10292.770933] LustreError: 323928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10292.912797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10293.017092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10298.495297] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10301.112188] LustreError: 325694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10301.125424] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10301.137184] LustreError: 325694:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10309.070820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10309.531241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10314.744462] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10315.050519] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10324.015803] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10325.861870] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10332.419589] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10333.866761] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10340.328907] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10340.346514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10340.377663] Lustre: Skipped 1 previous similar message [10340.381651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10340.403817] Lustre: Skipped 1 previous similar message [10345.611694] Lustre: server umount lustre-OST0000 complete [10350.564736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10350.576164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10350.576754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10350.593172] Lustre: Skipped 3 previous similar messages [10356.424824] Lustre: server umount lustre-MDT0000 complete [10360.405353] LustreError: 322077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834935 with bad export cookie 13827720234696284983 [10360.408779] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10360.417898] LustreError: 322077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10361.087691] Lustre: server umount lustre-MDT0001 complete [10373.629982] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [10376.703765] Key type lgssc unregistered [10377.112419] LNet: 328369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10377.140901] LNetError: 328369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10377.166935] LNet: Removed LNI 192.168.201.106@tcp [10377.930367] Key type .llcrypt unregistered [10377.932923] Key type ._llcrypt unregistered [10403.147802] Key type ._llcrypt registered [10403.149936] Key type .llcrypt registered [10403.243344] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [10404.303385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10404.489237] alg: No test for adler32 (adler32-zlib) [10405.519617] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [10405.797390] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [10407.455172] Key type lgssc registered [10408.332680] Lustre: Echo OBD driver; http://www.lustre.org/ [10416.004155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10416.019678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10417.569238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10421.783825] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10427.882321] LustreError: 329790:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10427.892777] LustreError: 329790:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10429.444240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10429.803847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10434.089648] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10441.111765] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10446.794521] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10450.402872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10450.409408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10450.435167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10452.454991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10452.457114] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10454.561688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10454.570281] Lustre: Skipped 2 previous similar messages [10459.616385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10459.623036] Lustre: Skipped 1 previous similar message [10464.223370] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10464.395488] Lustre: server umount lustre-MDT0000 complete [10464.739753] LustreError: 329789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10464.753063] LustreError: 329789:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10468.526952] LustreError: 329769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835043 with bad export cookie 2727924914347452216 [10468.532765] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10468.540889] LustreError: 329769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10468.844490] Lustre: server umount lustre-MDT0001 complete [10484.478031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10485.056710] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10485.112696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10485.431382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10490.499461] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10499.903824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10500.048881] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10500.099424] Lustre: 332377:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10500.582679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10507.620861] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10517.435966] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10524.561097] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10535.248100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10535.611857] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10535.620191] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10535.640263] LustreError: 334263:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10535.651423] LustreError: 334263:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10535.665057] LustreError: 334263:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10535.775632] Lustre: server umount lustre-OST0000 complete [10535.784455] LustreError: 334263:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10546.742388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10546.927355] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10547.109789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10552.735141] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10556.089735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10560.116402] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10567.530222] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10567.606388] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10567.703193] Lustre: lustre-OST0001: new disk, initializing [10567.708408] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10567.711221] Lustre: 335803:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10567.786251] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10573.134842] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10578.001153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10578.012152] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10578.094222] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10580.340662] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10583.857152] Lustre: server umount lustre-OST0001 complete [10588.640382] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10588.650704] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10588.655038] LustreError: Skipped 1 previous similar message [10588.670036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10588.679139] Lustre: Skipped 3 previous similar messages [10588.686800] LustreError: 335252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10588.709514] LustreError: 335252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10593.787323] LustreError: 334731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10593.805918] LustreError: 334731:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10594.471675] Lustre: server umount lustre-OST0000 complete [10598.889338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10598.905805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10598.914603] Lustre: Skipped 4 previous similar messages [10604.775947] Lustre: server umount lustre-MDT0000 complete [10608.893252] LustreError: 332370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835183 with bad export cookie 2727924914347452776 [10608.901791] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10608.911717] LustreError: 332370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10609.202213] Lustre: server umount lustre-MDT0001 complete [10620.398863] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [10622.989740] Key type lgssc unregistered [10623.240145] LNet: 337763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10623.247649] LNetError: 337763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10623.267350] LNet: Removed LNI 192.168.201.106@tcp [10623.990799] Key type .llcrypt unregistered [10623.993422] Key type ._llcrypt unregistered [10703.403982] Key type ._llcrypt registered [10703.409523] Key type .llcrypt registered [10703.552444] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [10718.672822] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [10719.767420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10719.791166] alg: No test for adler32 (adler32-zlib) [10720.911314] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [10721.124897] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [10722.743909] Key type lgssc registered [10723.663753] Lustre: Echo OBD driver; http://www.lustre.org/ [10757.434529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10757.447380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10758.695432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10758.744445] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10758.813503] Lustre: lustre-MDT0000: new disk, initializing [10758.901387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10758.914754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10763.182878] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10772.779022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10772.924799] Lustre: 342702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10772.978458] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10772.981274] Lustre: Skipped 1 previous similar message [10773.036967] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10773.058540] Lustre: lustre-MDT0001: new disk, initializing [10773.204761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10773.330620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10773.360201] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10777.144681] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10786.134064] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10791.855618] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10798.736760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10798.973600] Lustre: lustre-OST0000: new disk, initializing [10798.978933] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10798.987907] Lustre: 344672:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10799.094967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10805.176838] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10806.390938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10806.396148] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10806.506836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10814.365990] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10818.924814] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10819.081289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10823.804790] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10824.019246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10827.231951] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10827.233113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10827.237994] LustreError: Skipped 1 previous similar message [10827.245677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10827.249313] Lustre: Skipped 1 previous similar message [10827.267358] Lustre: Skipped 1 previous similar message [10832.094765] Lustre: server umount lustre-OST0000 complete [10834.921178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10834.937930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10834.946312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10837.477040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10837.490059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10842.596622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10842.599729] Lustre: Skipped 1 previous similar message [10849.247256] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10849.546464] Lustre: server umount lustre-MDT0000 complete [10853.389976] LustreError: 342696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835428 with bad export cookie 9584096723534421364 [10853.394536] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10853.404908] LustreError: 342696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10853.845734] Lustre: server umount lustre-MDT0001 complete [10862.898192] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 08:57:16 (1787835436) [10871.149446] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 08:57:24 (1787835444) [10879.584632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10880.041150] LustreError: 347428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10880.051686] LustreError: 347428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10880.132348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10884.078989] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10885.603773] LustreError: 347429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10889.699311] LustreError: 347428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10891.825139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10892.109467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10896.344375] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10903.555317] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10909.509767] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10916.315408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10916.773638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10922.836175] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10929.916663] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10938.600894] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10941.085874] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10943.244413] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10945.259291] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10947.638575] Lustre: Failing over lustre-MDT0000 [10948.246389] LustreError: 347425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10948.583056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10948.587893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10949.976805] Lustre: server umount lustre-MDT0000 complete [10953.322331] LustreError: 347423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10953.352154] LustreError: 347423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10963.553059] LustreError: 347425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10963.576102] LustreError: 347425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10966.754150] Lustre: 340928:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787835525/real 1787835525] req@000000000977c4f4 x1874681183409024/t0(0) o400->MGC192.168.201.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787835541 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10966.783254] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10970.527142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10976.225523] LustreError: 340926:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008edb5e42 x1874681183414400/t0(0) o250->MGC192.168.201.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10976.538837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10976.607853] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10978.915545] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10981.490587] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10981.877189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10981.898240] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10981.960902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10991.111778] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10992.735476] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10996.703598] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10998.812664] Lustre: Failing over lustre-MDT0000 [10999.108561] Lustre: server umount lustre-MDT0000 complete [10999.420296] LustreError: 349974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10999.444692] LustreError: 349974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [11002.338728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11002.352616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11002.378676] Lustre: Skipped 3 previous similar messages [11017.973242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11018.059605] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11018.284178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11018.326675] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11020.892335] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11021.609917] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11023.341908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11023.351064] Lustre: Skipped 2 previous similar messages [11023.372843] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [11023.449071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11030.942574] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11032.711595] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11043.813918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11043.816417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11043.829455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11043.863816] Lustre: Skipped 3 previous similar messages [11043.866897] Lustre: Skipped 3 previous similar messages [11045.215694] Lustre: server umount lustre-MDT0000 complete [11048.939612] LustreError: 347423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11048.955226] LustreError: 347423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [11049.265971] LustreError: 347410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835624 with bad export cookie 9584096723534424073 [11049.271959] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11049.273545] LustreError: 347410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11049.721349] Lustre: server umount lustre-MDT0001 complete [11064.102294] Lustre: server umount lustre-OST0000 complete [11081.390590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11087.070871] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11096.034344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11096.315073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11096.319272] Lustre: Skipped 1 previous similar message [11100.702223] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11108.029488] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11113.931777] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11120.975257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11121.128770] LustreError: 355603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11121.147303] LustreError: 355603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11126.418184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11130.050153] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11140.956991] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11147.236287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11147.237449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11147.272432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11147.278990] Lustre: Skipped 1 previous similar message [11148.908944] Lustre: server umount lustre-OST0000 complete [11158.987232] Lustre: server umount lustre-MDT0000 complete [11163.779970] LustreError: 353754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835738 with bad export cookie 9584096723534424822 [11163.784940] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11163.787432] LustreError: 353754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11164.109727] Lustre: server umount lustre-MDT0001 complete [11176.666350] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [11179.541947] Key type lgssc unregistered [11179.809591] LNet: 357317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11179.823945] LNetError: 357317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11179.852200] LNet: Removed LNI 192.168.201.106@tcp [11180.777363] Key type .llcrypt unregistered [11180.782904] Key type ._llcrypt unregistered [11208.128802] Key type ._llcrypt registered [11208.134264] Key type .llcrypt registered [11208.196205] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [11209.150323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11209.285318] alg: No test for adler32 (adler32-zlib) [11210.282755] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [11210.490953] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [11212.183171] Key type lgssc registered [11213.170836] Lustre: Echo OBD driver; http://www.lustre.org/ [11221.112630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11221.127216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11222.581340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11227.497650] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11232.742347] LustreError: 358839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11232.772535] LustreError: 358839:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11237.864137] LustreError: 358838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11237.970283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11238.544246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11243.599280] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11254.269337] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11261.783610] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11268.810545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11269.236488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11276.224062] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11284.327422] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11293.833790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11307.899470] Lustre: server umount lustre-MDT0000 complete [11309.537071] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11309.552794] Lustre: Skipped 2 previous similar messages [11309.556830] LustreError: 358838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11309.573236] LustreError: 358838:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11312.722484] LustreError: 358819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835887 with bad export cookie 2778332187505959992 [11312.733767] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11312.743973] LustreError: 358819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11313.174529] Lustre: server umount lustre-MDT0001 complete [11327.562597] Lustre: server umount lustre-OST0000 complete [11343.873617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11344.334638] LustreError: 362574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11344.423186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11349.364972] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11349.481191] LustreError: 362575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11359.732159] LustreError: 362575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11359.748727] LustreError: 362575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11360.076943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11360.508133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11366.467404] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11376.641578] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11383.165157] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11392.135079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11392.771965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11401.532642] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11412.756719] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11414.284553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11432.416634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11432.439131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11432.456522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11434.981612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11434.983941] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11436.302285] Lustre: server umount lustre-MDT0000 complete [11440.114844] LustreError: 362569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11440.156434] LustreError: 362569:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11441.250077] LustreError: 362554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836016 with bad export cookie 2778332187505961105 [11441.263184] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11441.272630] LustreError: 362554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11442.107696] Lustre: server umount lustre-MDT0001 complete [11457.500422] Lustre: server umount lustre-OST0000 complete [11477.974824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11478.745882] LustreError: 366488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11478.911892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11485.948988] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11498.391478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11503.902097] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11513.427725] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11520.114149] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11527.418286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11527.796230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11527.802470] Lustre: Skipped 1 previous similar message [11535.036232] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11543.223517] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11548.283243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11558.880726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11558.885682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11558.896399] Lustre: Skipped 1 previous similar message [11558.902353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11558.907328] Lustre: Skipped 2 previous similar messages [11560.972548] Lustre: server umount lustre-OST0000 complete [11565.536450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11565.540896] LustreError: Skipped 1 previous similar message [11565.546956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11565.570082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11565.577847] Lustre: Skipped 1 previous similar message [11571.928331] Lustre: server umount lustre-MDT0000 complete [11573.232290] LustreError: 366488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11573.266305] LustreError: 366488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11577.129558] LustreError: 366470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836152 with bad export cookie 2778332187505962127 [11577.132850] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11577.143087] LustreError: 366470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11577.714455] Lustre: server umount lustre-MDT0001 complete [11596.133996] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [11600.190650] Key type lgssc unregistered [11600.843159] LNet: 370029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11600.872714] LNetError: 370029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11600.921856] LNet: Removed LNI 192.168.201.106@tcp [11602.682968] Key type .llcrypt unregistered [11602.685813] Key type ._llcrypt unregistered [11635.560810] Key type ._llcrypt registered [11635.563691] Key type .llcrypt registered [11635.706163] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [11637.735618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11637.753443] alg: No test for adler32 (adler32-zlib) [11638.855205] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [11639.114179] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [11640.855228] Key type lgssc registered [11641.844505] Lustre: Echo OBD driver; http://www.lustre.org/ [11650.169712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11650.183626] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11650.194566] LustreError: 371471:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11650.214748] LustreError: 371471:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11650.224947] LustreError: 371471:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11650.235639] LustreError: 371471:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11650.246183] LustreError: 371471:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11659.050780] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11659.063876] LustreError: 371869:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11659.098837] LustreError: 371869:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11659.108054] LustreError: 371869:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11659.115188] LustreError: 371869:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11659.121804] LustreError: 371869:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11680.001510] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [11682.873187] Key type lgssc unregistered [11683.184202] LNet: 372876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11683.201265] LNetError: 372876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11683.222138] LNet: Removed LNI 192.168.201.106@tcp [11684.233505] Key type .llcrypt unregistered [11684.236337] Key type ._llcrypt unregistered [11703.817310] Key type ._llcrypt registered [11703.819364] Key type .llcrypt registered [11703.922603] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [11721.230257] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [11723.101202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11723.459210] alg: No test for adler32 (adler32-zlib) [11724.668777] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [11725.023891] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [11726.783174] Key type lgssc registered [11728.656690] Lustre: Echo OBD driver; http://www.lustre.org/ [11771.202307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11771.265471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11772.583925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11772.616411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11772.702478] Lustre: lustre-MDT0000: new disk, initializing [11772.805812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11772.822698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11777.982410] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11790.174609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11790.298644] Lustre: 376033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11790.390516] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11790.394535] Lustre: Skipped 1 previous similar message [11790.469723] Lustre: lustre-MDT0001: new disk, initializing [11790.603680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11790.714787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11790.727602] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11796.416264] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11807.676954] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11814.523387] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11821.781182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11822.116505] Lustre: lustre-OST0000: new disk, initializing [11822.120780] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11822.134220] Lustre: 378003:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11822.277754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11828.827271] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11828.833329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11828.840954] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11829.026716] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11839.426937] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11844.814700] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11845.134300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11849.014136] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11849.119481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11854.831775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11854.840044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11854.840534] LustreError: Skipped 1 previous similar message [11854.842842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11854.873022] Lustre: Skipped 1 previous similar message [11857.862975] Lustre: server umount lustre-OST0000 complete [11862.499879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11862.513383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11862.541193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11862.550776] Lustre: Skipped 1 previous similar message [11865.057150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11865.058676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11867.940942] Lustre: server umount lustre-MDT0000 complete [11872.534959] LustreError: 376830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836447 with bad export cookie 15254781870611261426 [11872.541402] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11872.556806] LustreError: 376830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11873.066045] Lustre: server umount lustre-MDT0001 complete [11883.806067] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11885.771818] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 09:14:18 (1787836458) [11894.207439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11894.835763] LustreError: 380465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11894.858445] LustreError: 380465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11894.927819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11900.395966] LustreError: 380466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11900.520679] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11904.484812] LustreError: 380465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11909.603117] LustreError: 380466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11910.348835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11910.880926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11916.321619] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11924.761116] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11931.905655] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11939.617354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11939.760473] LustreError: 382286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11940.081679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11947.201304] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11956.121727] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11963.724588] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11963.942650] Lustre: lustre-OST0001: new disk, initializing [11963.946905] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11963.953263] Lustre: 383305:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11964.065217] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11971.537317] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11973.244823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11973.256294] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11973.404903] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11983.067335] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11988.968781] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11988.970158] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11988.985332] Lustre: Skipped 1 previous similar message [11988.990360] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11989.007226] Lustre: Skipped 2 previous similar messages [11993.169438] Lustre: server umount lustre-OST0001 complete [11994.086688] LustreError: 382288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12012.001991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12012.007578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12012.010789] LustreError: Skipped 2 previous similar messages [12012.012692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12012.020614] Lustre: Skipped 1 previous similar message [12012.050869] Lustre: Skipped 2 previous similar messages [12014.563452] LustreError: 383738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12014.590165] LustreError: 383738:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [12024.827570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12024.845043] Lustre: Skipped 5 previous similar messages [12025.311335] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12025.488551] Lustre: server umount lustre-OST0000 complete [12029.922740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12029.940160] Lustre: Skipped 1 previous similar message [12044.255371] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12044.544977] Lustre: server umount lustre-MDT0000 complete [12049.270819] LustreError: 382310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836624 with bad export cookie 15254781870611262308 [12049.295051] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12049.297267] LustreError: 382310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12055.850272] Lustre: server umount lustre-MDT0001 complete [12069.703906] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [12072.807802] Key type lgssc unregistered [12073.232726] LNet: 385300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12073.238468] LNetError: 385300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12073.262931] LNet: Removed LNI 192.168.201.106@tcp [12074.431645] Key type .llcrypt unregistered [12074.433165] Key type ._llcrypt unregistered [12131.734216] Key type ._llcrypt registered [12131.735929] Key type .llcrypt registered [12131.854474] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [12133.389132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12133.530058] alg: No test for adler32 (adler32-zlib) [12134.524324] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [12134.824945] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [12136.520104] Key type lgssc registered [12137.646821] Lustre: Echo OBD driver; http://www.lustre.org/ [12145.772639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12145.827954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12147.138977] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12147.161399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12147.317340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12152.294865] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12161.681281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12161.743972] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12161.766932] Lustre: 388100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12162.035279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12167.007535] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12176.556416] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12183.720552] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12187.618284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12187.625547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12187.637182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12189.154296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12189.172936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12192.525800] Lustre: server umount lustre-MDT0000 complete [12196.247825] LustreError: 388091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836771 with bad export cookie 11968454341094173097 [12196.258328] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12196.263845] LustreError: 388091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12196.536810] Lustre: server umount lustre-MDT0001 complete [12205.970991] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 09:19:38 (1787836778) [12212.646215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12213.069920] LustreError: 390614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12213.084550] LustreError: 390614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12213.119268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12217.810178] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12218.340470] LustreError: 390615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12223.455992] LustreError: 390614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12226.579188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12226.792047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12230.942969] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12239.220974] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12245.415166] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12251.986974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12252.151731] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12252.342478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12258.749637] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12266.094449] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12273.502903] Lustre: Permanently deactivating lustre-OST0000 [12273.525305] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12280.412539] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12282.841756] LustreError: 393603:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12282.853796] LustreError: 393603:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12282.865310] LustreError: 393603:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12285.467217] LustreError: 393652:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12285.472882] LustreError: 393652:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12293.222898] Lustre: server umount lustre-OST0000 complete [12295.740535] LustreError: 393901:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12295.746281] LustreError: 393901:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12298.720869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12298.725877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12298.740494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12298.747306] Lustre: Skipped 1 previous similar message [12302.305602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12302.311498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12302.326670] Lustre: Skipped 1 previous similar message [12307.425191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12307.433821] Lustre: Skipped 1 previous similar message [12307.538756] Lustre: server umount lustre-MDT0000 complete [12311.632338] LustreError: 390595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836886 with bad export cookie 11968454341094173678 [12311.641560] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12311.654223] LustreError: 390595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12321.469283] Lustre: server umount lustre-MDT0001 complete [12330.282757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12334.628308] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12337.500255] LustreError: 395061:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12344.024273] Lustre: 395400:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.106@tcp [12346.321498] Lustre: server umount MGS complete [12358.430228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12358.826042] LustreError: 396126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12358.840497] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12358.849463] Lustre: Skipped 1 previous similar message [12358.891792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12363.902283] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12364.256480] LustreError: 396127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12373.670422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12374.033743] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12380.196761] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12391.253652] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12399.533226] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12401.433283] Lustre: Permanently reactivating lustre-OST0000 [12401.444403] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12401.448493] Lustre: Skipped 2 previous similar messages [12428.255152] LustreError: 397902:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12435.785951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12436.014878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12436.017944] Lustre: Skipped 1 previous similar message [12441.716453] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12449.255112] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12456.543043] Lustre: server umount lustre-OST0000 complete [12461.026377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12461.040938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12461.064395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12470.933154] Lustre: server umount lustre-MDT0000 complete [12473.767163] Lustre: 387425:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787837032/real 1787837032] req@00000000d48788c5 x1874682665076608/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787837048 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12473.826776] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12473.840048] LustreError: 396126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12473.842626] Lustre: Skipped 1 previous similar message [12473.879651] LustreError: 396126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12474.849359] Lustre: 387423:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787837033/real 1787837033] req@000000003c464a17 x1874682665077376/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787837049 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12475.148788] LustreError: 396107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837050 with bad export cookie 11968454341094174728 [12475.150717] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12475.159272] LustreError: 396107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12475.786890] Lustre: server umount lustre-MDT0001 complete [12488.442906] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [12491.037408] Key type lgssc unregistered [12491.289532] LNet: 400492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12491.296693] LNetError: 400492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12491.315677] LNet: Removed LNI 192.168.201.106@tcp [12492.186253] Key type .llcrypt unregistered [12492.190641] Key type ._llcrypt unregistered [12510.585727] Key type ._llcrypt registered [12510.588330] Key type .llcrypt registered [12510.665789] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [12525.750488] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [12526.832669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12526.906298] alg: No test for adler32 (adler32-zlib) [12527.992939] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [12528.172179] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [12529.839898] Key type lgssc registered [12530.884886] Lustre: Echo OBD driver; http://www.lustre.org/ [12569.057841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12569.112156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12570.421876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12570.477247] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12570.546895] Lustre: lustre-MDT0000: new disk, initializing [12570.625606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12570.640261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12575.555881] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12587.992223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12588.184614] Lustre: 403649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12588.255177] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12588.262503] Lustre: Skipped 1 previous similar message [12588.382596] Lustre: lustre-MDT0001: new disk, initializing [12588.524633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12588.576507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12588.592270] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12594.805307] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12606.175487] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12611.754856] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12617.639237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12617.889482] Lustre: lustre-OST0000: new disk, initializing [12617.892333] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12617.899199] Lustre: 405620:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12617.981439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12623.402605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12623.416531] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12623.513592] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12623.634912] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12632.355988] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12637.847347] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12638.056457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12642.343941] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12642.549599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12649.452311] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12649.457549] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12649.469610] LustreError: Skipped 1 previous similar message [12649.473838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12649.482383] Lustre: Skipped 1 previous similar message [12649.516360] Lustre: Skipped 1 previous similar message [12650.678709] Lustre: server umount lustre-OST0000 complete [12654.561887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12654.564589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12654.573219] Lustre: Skipped 1 previous similar message [12654.586394] Lustre: Skipped 1 previous similar message [12659.680781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12659.683186] Lustre: Skipped 1 previous similar message [12660.590837] Lustre: server umount lustre-MDT0000 complete [12664.246934] LustreError: 403641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837239 with bad export cookie 6617937225571424811 [12664.251237] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12664.261147] LustreError: 403641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12664.578528] Lustre: server umount lustre-MDT0001 complete [12673.127894] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12674.931737] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 09:27:28 (1787837248) [12681.360199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12681.783677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12685.907740] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12686.826698] LustreError: 408082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12686.845395] LustreError: 408082:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12691.936205] LustreError: 408083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12694.076420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12694.471139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12699.044361] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12707.234680] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12713.858549] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12720.648467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12720.989831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12726.385638] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12733.493366] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12735.552149] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12738.530955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12750.817257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12750.823196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12750.838193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12750.839570] Lustre: Skipped 1 previous similar message [12750.846643] Lustre: Skipped 1 previous similar message [12755.508460] Lustre: server umount lustre-OST0000 complete [12761.057693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12761.062495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12761.082708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12761.089364] Lustre: Skipped 1 previous similar message [12761.103842] Lustre: Skipped 1 previous similar message [12770.275695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12770.288738] Lustre: Skipped 3 previous similar messages [12773.346565] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12773.594850] Lustre: server umount lustre-MDT0000 complete [12775.397596] LustreError: 408079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12775.414191] LustreError: 408079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12777.314029] LustreError: 408063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837352 with bad export cookie 6617937225571425693 [12777.318683] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12777.333501] LustreError: 408063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12777.731077] Lustre: server umount lustre-MDT0001 complete [12787.474561] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [12790.008108] Key type lgssc unregistered [12790.330900] LNet: 411811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12790.336855] LNetError: 411811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12791.401994] LNet: Removed LNI 192.168.201.106@tcp [12792.244736] Key type .llcrypt unregistered [12792.247154] Key type ._llcrypt unregistered [12820.802521] Key type ._llcrypt registered [12820.808257] Key type .llcrypt registered [12820.944224] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [12821.463341] Key type .llcrypt unregistered [12821.469230] Key type ._llcrypt unregistered [12841.239451] Key type ._llcrypt registered [12841.240503] Key type .llcrypt registered [12841.319335] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [12842.775123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12842.847435] alg: No test for adler32 (adler32-zlib) [12843.939105] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [12844.186446] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [12845.855226] Key type lgssc registered [12847.243393] Lustre: Echo OBD driver; http://www.lustre.org/ [12854.870269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12854.879338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12856.300587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12860.046507] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12866.533952] LustreError: 414026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12866.549857] LustreError: 414026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12867.185925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12867.419296] LustreError: 414027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12867.438524] LustreError: 414027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12867.458764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12872.438920] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12872.679833] LustreError: 414026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12872.688918] LustreError: 414026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12877.793214] LustreError: 414027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12877.811261] LustreError: 414027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12879.134816] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12881.901184] LustreError: 414026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12881.916404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12881.922512] LustreError: 414026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12885.548124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12885.960286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12890.602134] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12901.356810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12901.367159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12901.384904] Lustre: Skipped 1 previous similar message [12901.385363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12906.472376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12906.480982] Lustre: Skipped 1 previous similar message [12911.590110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12911.606561] Lustre: Skipped 1 previous similar message [12913.121551] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12913.331264] Lustre: server umount lustre-OST0000 complete [12921.826182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12921.827968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12921.836023] Lustre: Skipped 1 previous similar message [12921.858290] Lustre: Skipped 2 previous similar messages [12923.235233] Lustre: server umount lustre-MDT0000 complete [12926.945207] LustreError: 414027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12926.953657] LustreError: 414027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12927.009278] LustreError: 414008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837501 with bad export cookie 14818274717107740545 [12927.023740] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12927.326994] Lustre: server umount lustre-MDT0001 complete [12939.246767] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [12942.580743] Key type lgssc unregistered [12942.991836] LNet: 417176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12943.008995] LNetError: 417176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12944.039235] LNet: Removed LNI 192.168.201.106@tcp [12944.962822] Key type .llcrypt unregistered [12944.965172] Key type ._llcrypt unregistered [12974.501940] Key type ._llcrypt registered [12974.503981] Key type .llcrypt registered [12974.655932] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [12975.795838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12975.904967] alg: No test for adler32 (adler32-zlib) [12976.976147] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [12977.205765] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [12978.887235] Key type lgssc registered [12979.858677] Lustre: Echo OBD driver; http://www.lustre.org/ [12989.349163] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12989.385106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13004.898205] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13010.080212] LustreError: 418572:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.106@tcp: failed processing log, type 4: rc = -110 [13036.831345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13043.096541] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13050.217469] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13056.791284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13057.273288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13057.344166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13061.199126] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13062.630575] LustreError: 418598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13062.659287] LustreError: 418598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13067.745379] LustreError: 418597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13069.861711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13070.293484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13075.095631] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13082.507654] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13088.707207] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13096.945309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13096.972701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13096.981568] Lustre: Skipped 1 previous similar message [13098.652674] Lustre: server umount lustre-OST0000 complete [13103.271949] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13103.281181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13103.289950] Lustre: Skipped 1 previous similar message [13103.297465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13103.781245] Lustre: server umount lustre-MDT0000 complete [13107.169812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13107.172270] LustreError: 419613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13107.200197] LustreError: 419613:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13107.722706] LustreError: 418578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837682 with bad export cookie 4045277229368891834 [13107.729279] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13107.743581] LustreError: 418578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13108.287981] Lustre: server umount lustre-MDT0001 complete [13122.445450] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [13125.701778] Key type lgssc unregistered [13126.036942] LNet: 422144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13126.046709] LNetError: 422144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13126.074366] LNet: Removed LNI 192.168.201.106@tcp [13126.989414] Key type .llcrypt unregistered [13126.991695] Key type ._llcrypt unregistered [13157.166864] Key type ._llcrypt registered [13157.169614] Key type .llcrypt registered [13157.305712] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [13158.629238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13158.718105] alg: No test for adler32 (adler32-zlib) [13160.106914] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [13160.541699] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [13162.208267] Key type lgssc registered [13163.507994] Lustre: Echo OBD driver; http://www.lustre.org/ [13172.840460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13172.861564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13174.441926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13180.263039] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13185.009847] LustreError: 423571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13185.037670] LustreError: 423571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13190.112601] LustreError: 423572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13190.169281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13190.611598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13196.287607] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13204.843675] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13212.063242] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13220.956968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13221.476845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13229.427976] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13238.380321] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13245.109356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13254.783088] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.6@tcp (stopping) [13255.650803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13255.653143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13255.680173] Lustre: Skipped 2 previous similar messages [13255.697416] Lustre: Skipped 2 previous similar messages [13259.081872] Lustre: server umount lustre-MDT0000 complete [13259.868455] LustreError: 423566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13261.833854] LustreError: 423552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.6@tcp arrived at 1787837836 with bad export cookie 12496971180311759173 [13261.862635] LustreError: 423552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13264.403823] LustreError: 423550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837839 with bad export cookie 12496971180311758501 [13264.418255] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13264.428280] LustreError: 423550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13265.889721] LustreError: 423572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13265.891210] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13265.908107] LustreError: 423572:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13265.936914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13269.923764] LustreError: 424859:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13269.930102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13269.975995] LustreError: 424859:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13270.958947] Lustre: server umount lustre-OST0000 complete [13279.862547] Lustre: server umount lustre-MDT0001 complete [13300.568173] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [13305.404949] Key type lgssc unregistered [13305.894796] LNet: 427305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13305.898849] LNetError: 427305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13305.927842] LNet: Removed LNI 192.168.201.106@tcp [13307.436881] Key type .llcrypt unregistered [13307.438472] Key type ._llcrypt unregistered [13339.349090] Key type ._llcrypt registered [13339.350791] Key type .llcrypt registered [13339.523680] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [13341.376551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13341.393433] alg: No test for adler32 (adler32-zlib) [13342.611907] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [13342.974224] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [13344.703308] Key type lgssc registered [13346.136700] Lustre: Echo OBD driver; http://www.lustre.org/ [13354.839285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13354.864415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13356.642047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13361.547144] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13366.249704] LustreError: 428723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13366.293675] LustreError: 428723:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13371.159738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13371.364533] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13371.579267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13376.651892] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13385.820073] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13393.258571] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13401.083525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13401.461496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13402.484846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13408.019631] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13417.014494] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13423.079192] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13423.090118] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13423.114360] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13424.754446] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.6@tcp (stopping) [13425.388068] Lustre: server umount lustre-MDT0001 complete [13427.178921] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13427.179396] LustreError: 431215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13427.237340] LustreError: 431215:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13432.288832] LustreError: 428724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13432.316285] LustreError: 428724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13433.315806] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13433.324423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13433.375474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13437.408085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13437.409742] LustreError: 431215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13439.041535] Lustre: server umount lustre-OST0000 complete [13442.533681] LustreError: 428723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13444.371591] Lustre: server umount lustre-MDT0000 complete [13460.258586] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [13463.262369] Key type lgssc unregistered [13463.627354] LNet: 432414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13463.639297] LNetError: 432414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13463.668372] LNet: Removed LNI 192.168.201.106@tcp [13464.718963] Key type .llcrypt unregistered [13464.721782] Key type ._llcrypt unregistered [13495.349453] Key type ._llcrypt registered [13495.353095] Key type .llcrypt registered [13495.539312] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [13514.509519] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [13515.958265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13516.275295] alg: No test for adler32 (adler32-zlib) [13517.543620] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [13517.960790] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [13519.679896] Key type lgssc registered [13520.953742] Lustre: Echo OBD driver; http://www.lustre.org/ [13565.638855] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [13582.697349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13582.739566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13583.990883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13584.010383] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13584.077429] Lustre: lustre-MDT0000: new disk, initializing [13584.176089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13584.198156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13589.726610] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13607.964412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13608.280839] Lustre: lustre-OST0000: new disk, initializing [13608.286474] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13608.294215] Lustre: Skipped 1 previous similar message [13608.300763] Lustre: 437452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13608.361797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13615.253082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13615.261341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13615.313261] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13616.270289] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13628.907339] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13646.879816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13647.064791] Lustre: 436436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13647.166723] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13647.291388] Lustre: lustre-MDT0001: new disk, initializing [13647.482329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13647.537626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13647.553944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13652.316322] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13662.920433] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13663.156124] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13667.835869] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13668.043300] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13677.169408] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.6@tcp (stopping) [13678.053117] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13678.063560] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13678.068126] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13678.081613] Lustre: Skipped 1 previous similar message [13678.090270] Lustre: Skipped 1 previous similar message [13679.663195] Lustre: server umount lustre-MDT0001 complete [13688.293474] LustreError: 438307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13688.322857] LustreError: 438307:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13693.414031] LustreError: 436442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13693.442194] LustreError: 436442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13698.547948] LustreError: 438307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13698.588763] LustreError: 438307:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13702.624779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13702.627757] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13702.644754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13703.650440] LustreError: 436447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13705.577200] Lustre: server umount lustre-OST0000 complete [13708.770105] LustreError: 436446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13716.754371] Lustre: server umount lustre-MDT0000 complete [13732.204725] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [13735.328209] Key type lgssc unregistered [13735.682446] LNet: 441205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13735.687607] LNetError: 441205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13736.746232] LNet: Removed LNI 192.168.201.106@tcp [13737.693682] Key type .llcrypt unregistered [13737.696066] Key type ._llcrypt unregistered [13784.571053] Key type ._llcrypt registered [13784.573386] Key type .llcrypt registered [13784.685460] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [13785.047765] Key type .llcrypt unregistered [13785.049286] Key type ._llcrypt unregistered [13805.669948] Key type ._llcrypt registered [13805.677540] Key type .llcrypt registered [13805.812069] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [13807.746024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13807.804195] alg: No test for adler32 (adler32-zlib) [13809.106678] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [13809.388498] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [13811.072991] Key type lgssc registered [13812.459562] Lustre: Echo OBD driver; http://www.lustre.org/ [13843.549853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13843.574458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13844.854583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13844.891600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13844.983460] Lustre: lustre-MDT0000: new disk, initializing [13845.067660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13845.087807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13849.769841] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13861.397137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13861.570248] Lustre: 444589:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13861.685267] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13861.694047] Lustre: Skipped 1 previous similar message [13861.837787] Lustre: lustre-MDT0001: new disk, initializing [13861.974541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13862.026673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13862.047317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13867.366173] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13878.364109] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13885.061478] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13893.026922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13893.379306] Lustre: lustre-OST0000: new disk, initializing [13893.387352] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13893.394863] Lustre: 446560:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13893.538851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13899.870286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13899.894272] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13900.089862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13900.921276] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13911.325585] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13920.736982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13920.743299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13920.747293] Lustre: Skipped 1 previous similar message [13925.963590] Lustre: server umount lustre-MDT0000 complete [13930.489327] LustreError: 446559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838505 with bad export cookie 9555687657723929980 [13930.498836] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13930.516531] LustreError: 446559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13930.979645] LustreError: 444597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13930.979990] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13931.000547] LustreError: 444597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13931.024176] Lustre: Skipped 1 previous similar message [13931.162787] Lustre: server umount lustre-MDT0001 complete [13936.533212] Lustre: server umount lustre-OST0000 complete [13952.617658] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 09:48:44 (1787838524) [13963.156389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13963.792324] LustreError: 448588:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13963.887224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13968.866090] LustreError: 448589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13969.584837] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13973.989822] LustreError: 448588:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13979.110882] LustreError: 448589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13979.557472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13980.041663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13985.463661] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13995.036852] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14002.345585] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14024.373353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14024.568642] Lustre: Found index 0 for lustre-OST0000, updating log [14024.736953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14032.394796] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [14034.995629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14040.227888] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14047.888840] Lustre: server umount lustre-OST0000 complete [14065.716092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14065.724475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14065.736510] Lustre: Skipped 1 previous similar message [14065.751181] Lustre: Skipped 3 previous similar messages [14066.789213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14072.290077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14072.302227] Lustre: Skipped 2 previous similar messages [14073.404142] Lustre: server umount lustre-MDT0000 complete [14077.400668] LustreError: 448569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838652 with bad export cookie 9555687657723931373 [14077.408632] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14077.415977] LustreError: 448569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14077.765946] Lustre: server umount lustre-MDT0001 complete [14085.919173] LNet: 1 local NIs in recovery (showing 1): 192.168.201.106@tcp [14087.224834] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 09:51:00 (1787838660) [14711.102485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14711.723262] LustreError: 455011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14711.809059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14716.233128] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [14716.896984] LustreError: 455012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14722.019107] LustreError: 455011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14725.719634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14726.030791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14730.943636] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [14738.716148] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14745.414748] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14763.871349] LNet: 1 local NIs in recovery (showing 1): 192.168.201.106@tcp [14766.513587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14766.920791] Lustre: Found index 0 for lustre-OST0000, updating log [14767.256108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14775.613595] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [14784.566515] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14787.686791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14787.719666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14792.947960] Lustre: server umount lustre-OST0000 complete [14797.793593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14797.808123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14797.847271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14798.885107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14798.887051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14800.993053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14800.995358] Lustre: Skipped 3 previous similar messages [14804.065844] LustreError: 455012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14804.089050] LustreError: 455012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14805.799041] Lustre: server umount lustre-MDT0000 complete [14810.751797] LustreError: 454993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839385 with bad export cookie 9555687657723932444 [14810.766400] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14811.680758] Lustre: server umount lustre-MDT0001 complete [14823.263378] LNet: 1 local NIs in recovery (showing 1): 192.168.201.106@tcp [14843.241623] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [14853.553631] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [14900.526849] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 10:04:33 (1787839473) [14921.818601] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [14924.992990] Key type lgssc unregistered [14925.331608] LNet: 461631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14925.347576] LNetError: 461631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15029.796800] LNet: Removed LNI 192.168.201.106@tcp [15030.629699] Key type .llcrypt unregistered [15030.632058] Key type ._llcrypt unregistered [15053.806384] Key type ._llcrypt registered [15053.814883] Key type .llcrypt registered [15054.000419] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [15055.476253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15055.588972] alg: No test for adler32 (adler32-zlib) [15057.139674] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [15057.608138] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [15059.471814] Key type lgssc registered [15061.573961] Lustre: Echo OBD driver; http://www.lustre.org/ [15068.034796] LNet: Added LNI 192.168.201.106@tcp100 [8/256/0/180] [15069.111131] LNet: Added LNI 192.168.201.106@tcp101 [8/256/0/180] [15070.085062] LNet: Added LNI 192.168.201.106@tcp102 [8/256/0/180] [15071.273010] LNet: Added LNI 192.168.201.106@tcp103 [8/256/0/180] [15072.651680] LNet: Added LNI 192.168.201.106@tcp104 [8/256/0/180] [15074.212034] LNet: Added LNI 192.168.201.106@tcp105 [8/256/0/180] [15075.095897] LNet: Added LNI 192.168.201.106@tcp106 [8/256/0/180] [15076.172572] LNet: Added LNI 192.168.201.106@tcp107 [8/256/0/180] [15077.124799] LNet: Added LNI 192.168.201.106@tcp108 [8/256/0/180] [15078.364653] LNet: Added LNI 192.168.201.106@tcp109 [8/256/0/180] [15079.263722] LNet: Added LNI 192.168.201.106@tcp110 [8/256/0/180] [15080.383380] LNet: Added LNI 192.168.201.106@tcp111 [8/256/0/180] [15081.400311] LNet: Added LNI 192.168.201.106@tcp112 [8/256/0/180] [15082.275615] LNet: Added LNI 192.168.201.106@tcp113 [8/256/0/180] [15083.194793] LNet: Added LNI 192.168.201.106@tcp114 [8/256/0/180] [15084.089555] LNet: Added LNI 192.168.201.106@tcp115 [8/256/0/180] [15084.920486] LNet: Added LNI 192.168.201.106@tcp116 [8/256/0/180] [15085.909755] LNet: Added LNI 192.168.201.106@tcp117 [8/256/0/180] [15086.887661] LNet: Added LNI 192.168.201.106@tcp118 [8/256/0/180] [15087.636681] LNet: Added LNI 192.168.201.106@tcp119 [8/256/0/180] [15088.438262] LNet: Added LNI 192.168.201.106@tcp120 [8/256/0/180] [15089.334568] LNet: Added LNI 192.168.201.106@tcp121 [8/256/0/180] [15090.151122] LNet: Added LNI 192.168.201.106@tcp122 [8/256/0/180] [15090.952936] LNet: Added LNI 192.168.201.106@tcp123 [8/256/0/180] [15091.710951] LNet: Added LNI 192.168.201.106@tcp124 [8/256/0/180] [15092.622686] LNet: Added LNI 192.168.201.106@tcp125 [8/256/0/180] [15093.537692] LNet: Added LNI 192.168.201.106@tcp126 [8/256/0/180] [15094.562697] LNet: Added LNI 192.168.201.106@tcp127 [8/256/0/180] [15095.660318] LNet: Added LNI 192.168.201.106@tcp128 [8/256/0/180] [15096.689499] LNet: Added LNI 192.168.201.106@tcp129 [8/256/0/180] [15097.634442] LNet: Added LNI 192.168.201.106@tcp130 [8/256/0/180] [15098.578530] LNet: Added LNI 192.168.201.106@tcp131 [8/256/0/180] [15099.325283] LNet: Added LNI 192.168.201.106@tcp132 [8/256/0/180] [15100.308890] LNet: Added LNI 192.168.201.106@tcp133 [8/256/0/180] [15101.245996] LNet: Added LNI 192.168.201.106@tcp134 [8/256/0/180] [15102.086563] LNet: Added LNI 192.168.201.106@tcp135 [8/256/0/180] [15108.749160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15108.775102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15109.923937] Lustre: 464662:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.106@tcp: can't add NID 192.168.201.106@tcp130: rc = -75 [15110.016397] LNetError: 462119:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.106 [15110.035967] LNetError: Protocol error connecting to 192.168.201.106@tcp119 at host 192.168.201.106:988: is it running a compatible version of Lustre? [15110.062071] LNetError: Skipped 9 previous similar messages [15110.218627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15110.265765] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15110.339800] Lustre: lustre-MDT0000: new disk, initializing [15110.460726] Lustre: 464718:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.106@tcp: can't add NID 192.168.201.106@tcp130: rc = -75 [15110.471923] Lustre: 464718:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15110.494250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15110.540609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15110.884604] LNet: 9 local NIs in recovery (showing 9): 192.168.201.106@tcp124, 192.168.201.106@tcp125, 192.168.201.106@tcp123, 192.168.201.106@tcp127, 192.168.201.106@tcp128, 192.168.201.106@tcp129, 192.168.201.106@tcp126, 192.168.201.106@tcp119, 192.168.201.106@tcp117 [15111.072487] LNetError: 462119:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.106 [15111.081955] LNetError: 462119:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [15111.103850] LNetError: Protocol error connecting to 192.168.201.106@tcp118 at host 192.168.201.106:988: is it running a compatible version of Lustre? [15111.133286] LNetError: Skipped 1 previous similar message [15114.020967] LNetError: 462121:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.106 [15114.037311] LNetError: 462121:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [15114.053095] LNetError: Protocol error connecting to 192.168.201.106@tcp124 at host 192.168.201.106:988: is it running a compatible version of Lustre? [15114.070311] LNetError: Skipped 12 previous similar messages [15115.770615] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [15118.113160] LNetError: 462120:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.106 [15118.121177] LNetError: 462120:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [15118.127964] LNetError: Protocol error connecting to 192.168.201.106@tcp124 at host 192.168.201.106:988: is it running a compatible version of Lustre? [15118.135942] LNetError: Skipped 1 previous similar message [15126.304980] LNetError: 464683:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.106 [15126.314938] LNetError: Protocol error connecting to 192.168.201.106@tcp124 at host 192.168.201.106:988: is it running a compatible version of Lustre? [15127.360480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15127.479467] Lustre: 464677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15127.529554] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15127.531938] Lustre: Skipped 1 previous similar message [15127.576893] Lustre: lustre-MDT0001: new disk, initializing [15127.613049] Lustre: 465481:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.106@tcp: can't add NID 192.168.201.106@tcp130: rc = -75 [15127.618986] Lustre: 465481:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15127.783329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15127.835136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15127.843470] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15132.770758] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [15134.771763] Lustre: 465823:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.106@tcp: can't add NID 192.168.201.106@tcp130: rc = -75 [15134.781887] Lustre: 465823:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15140.640561] LNetError: 464682:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.106 [15140.669105] LNetError: Protocol error connecting to 192.168.201.106@tcp120 at host 192.168.201.106:988: is it running a compatible version of Lustre? [15144.915583] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15151.480510] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15157.025186] LNetError: 462121:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.106 [15157.032934] LNetError: 462121:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 13 previous similar messages [15157.063329] LNetError: Protocol error connecting to 192.168.201.106@tcp126 at host 192.168.201.106:988: is it running a compatible version of Lustre? [15157.086295] LNetError: Skipped 13 previous similar messages [15159.244357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15159.513817] Lustre: lustre-OST0000: new disk, initializing [15159.517310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15159.526330] Lustre: 466654:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15159.615150] Lustre: 466659:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.106@tcp: can't add NID 192.168.201.106@tcp130: rc = -75 [15159.624055] Lustre: 466659:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15159.672401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15165.992152] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [15167.532262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15167.543568] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15167.691599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15170.335360] LNet: 1 local NIs in recovery (showing 1): 192.168.201.106@tcp124 [15175.445905] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15188.467109] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15188.476463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15188.493271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15190.816632] LNetError: 462119:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.106 [15190.836970] LNetError: 462119:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 23 previous similar messages [15190.858306] LNetError: Protocol error connecting to 192.168.201.106@tcp117 at host 192.168.201.106:988: is it running a compatible version of Lustre? [15190.878346] LNetError: Skipped 23 previous similar messages [15191.033407] Lustre: server umount lustre-OST0000 complete [15198.690919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15198.697936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15198.737218] Lustre: Skipped 2 previous similar messages [15198.777019] Lustre: Skipped 2 previous similar messages [15203.514363] Lustre: server umount lustre-MDT0000 complete [15207.816976] LustreError: 465480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839782 with bad export cookie 5739776458923278512 [15207.821376] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15207.838287] LustreError: 465480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15208.356254] Lustre: server umount lustre-MDT0001 complete [15225.420674] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [15229.686701] Key type lgssc unregistered [15230.396914] LNet: 468572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15230.413505] LNetError: 468572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15230.433333] LNet: Removed LNI 192.168.201.106@tcp [15231.457277] LNet: Removed LNI 192.168.201.106@tcp100 [15231.461567] LNetError: 468572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15231.477236] LNetError: 468572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15231.493627] LNet: Removed LNI 192.168.201.106@tcp101 [15232.544015] LNet: Removed LNI 192.168.201.106@tcp102 [15232.553549] LNetError: 468572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15232.580132] LNetError: 468572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15232.596712] LNet: Removed LNI 192.168.201.106@tcp103 [15233.631273] LNet: Removed LNI 192.168.201.106@tcp104 [15234.655359] LNet: Removed LNI 192.168.201.106@tcp105 [15234.691592] LNetError: 468572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15234.707378] LNetError: 468572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15234.754066] LNet: Removed LNI 192.168.201.106@tcp106 [15235.808736] LNet: Removed LNI 192.168.201.106@tcp107 [15236.831368] LNet: Removed LNI 192.168.201.106@tcp108 [15237.855325] LNet: Removed LNI 192.168.201.106@tcp109 [15238.879408] LNet: Removed LNI 192.168.201.106@tcp110 [15238.891168] LNetError: 468572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15238.900226] LNetError: 468572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15238.911169] LNet: Removed LNI 192.168.201.106@tcp111 [15239.968089] LNet: Removed LNI 192.168.201.106@tcp112 [15240.992695] LNet: Removed LNI 192.168.201.106@tcp113 [15242.015243] LNet: Removed LNI 192.168.201.106@tcp114 [15243.041239] LNet: Removed LNI 192.168.201.106@tcp115 [15244.063225] LNet: Removed LNI 192.168.201.106@tcp116 [15245.087301] LNet: Removed LNI 192.168.201.106@tcp117 [15246.111527] LNet: Removed LNI 192.168.201.106@tcp118 [15247.135634] LNet: Removed LNI 192.168.201.106@tcp119 [15247.150484] LNetError: 468572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15247.162497] LNetError: 468572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15247.180057] LNet: Removed LNI 192.168.201.106@tcp120 [15248.223403] LNet: Removed LNI 192.168.201.106@tcp121 [15249.247308] LNet: Removed LNI 192.168.201.106@tcp122 [15250.271405] LNet: Removed LNI 192.168.201.106@tcp123 [15250.277238] LNet: Removed LNI 192.168.201.106@tcp124 [15251.295147] LNet: Removed LNI 192.168.201.106@tcp125 [15252.319419] LNet: Removed LNI 192.168.201.106@tcp126 [15253.343346] LNet: Removed LNI 192.168.201.106@tcp127 [15254.367389] LNet: Removed LNI 192.168.201.106@tcp128 [15255.391277] LNet: Removed LNI 192.168.201.106@tcp129 [15255.393624] LNet: Removed LNI 192.168.201.106@tcp130 [15255.395379] LNet: Removed LNI 192.168.201.106@tcp131 [15255.397812] LNet: Removed LNI 192.168.201.106@tcp132 [15255.400307] LNet: Removed LNI 192.168.201.106@tcp133 [15255.403369] LNet: Removed LNI 192.168.201.106@tcp134 [15255.416176] LNet: Removed LNI 192.168.201.106@tcp135 [15256.270442] Key type .llcrypt unregistered [15256.273115] Key type ._llcrypt unregistered [15283.448594] Key type ._llcrypt registered [15283.451524] Key type .llcrypt registered [15283.587821] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [15298.797288] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [15300.204921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15300.548280] alg: No test for adler32 (adler32-zlib) [15301.841274] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [15302.153172] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [15303.918184] Key type lgssc registered [15305.283311] Lustre: Echo OBD driver; http://www.lustre.org/ [15345.993212] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 10:11:58 (1787839918) [15367.068972] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [15370.862332] Key type lgssc unregistered [15371.200810] LNet: 472665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15371.209726] LNetError: 472665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15371.228262] LNet: Removed LNI 192.168.201.106@tcp [15371.961581] Key type .llcrypt unregistered [15371.963371] Key type ._llcrypt unregistered [15393.439842] Key type ._llcrypt registered [15393.448278] Key type .llcrypt registered [15393.605784] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [15394.948391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15395.121601] alg: No test for adler32 (adler32-zlib) [15396.137453] Lustre: Lustre: Build Version: 2.17.57_69_gb81e743 [15396.489620] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [15398.191740] Key type lgssc registered [15400.085917] Lustre: Echo OBD driver; http://www.lustre.org/ [15413.218939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15413.255290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15414.799707] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15414.907773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15415.115096] Lustre: lustre-MDT0000: new disk, initializing [15415.226080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15415.271888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15420.328247] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [15432.669921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15432.823619] Lustre: 473881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15432.939472] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15432.943395] Lustre: Skipped 1 previous similar message [15433.045710] Lustre: lustre-MDT0001: new disk, initializing [15433.151121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15433.232532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15433.248697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15438.900451] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [15449.390066] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15456.095651] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15464.342148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15464.677478] Lustre: lustre-OST0000: new disk, initializing [15464.686835] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15464.704868] Lustre: 475849:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15464.829642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15472.304590] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [15474.809640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15474.844722] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15474.992522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15483.153932] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15487.877470] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing lnet_if_list [15493.552158] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15493.586350] LNet: Added LNI 192.168.201.106@tcp10 [8/256/0/180] [15499.919365] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15499.958465] LNet: Added LNI 192.168.201.106@tcp11 [8/256/0/180] [15503.949880] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15503.992383] LNet: Added LNI 192.168.201.106@tcp12 [8/256/0/180] [15508.437796] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15508.474552] LNet: Added LNI 192.168.201.106@tcp13 [8/256/0/180] [15512.775398] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15512.799435] LNet: Added LNI 192.168.201.106@tcp14 [8/256/0/180] [15517.652796] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15517.717957] LNet: Added LNI 192.168.201.106@tcp15 [8/256/0/180] [15523.825802] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15523.869635] LNet: Added LNI 192.168.201.106@tcp16 [8/256/0/180] [15529.961451] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15530.005882] LNet: Added LNI 192.168.201.106@tcp17 [8/256/0/180] [15536.585545] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15536.627560] LNet: Added LNI 192.168.201.106@tcp18 [8/256/0/180] [15542.568231] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15542.605147] LNet: Added LNI 192.168.201.106@tcp19 [8/256/0/180] [15548.318886] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15548.353406] LNet: Added LNI 192.168.201.106@tcp20 [8/256/0/180] [15553.710627] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15553.734251] LNet: Added LNI 192.168.201.106@tcp21 [8/256/0/180] [15559.519942] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15559.556349] LNet: Added LNI 192.168.201.106@tcp22 [8/256/0/180] [15564.316703] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15564.342103] LNet: Added LNI 192.168.201.106@tcp23 [8/256/0/180] [15569.329586] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15569.372809] LNet: Added LNI 192.168.201.106@tcp24 [8/256/0/180] [15574.805502] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15574.837464] LNet: Added LNI 192.168.201.106@tcp25 [8/256/0/180] [15580.706426] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15580.727211] LNet: Added LNI 192.168.201.106@tcp26 [8/256/0/180] [15587.558892] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15587.588125] LNet: Added LNI 192.168.201.106@tcp27 [8/256/0/180] [15593.322768] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15593.344596] LNet: Added LNI 192.168.201.106@tcp28 [8/256/0/180] [15599.544215] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15599.578857] LNet: Added LNI 192.168.201.106@tcp29 [8/256/0/180] [15606.334739] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15606.367265] LNet: Added LNI 192.168.201.106@tcp30 [8/256/0/180] [15613.009797] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15613.060624] LNet: Added LNI 192.168.201.106@tcp31 [8/256/0/180] [15619.804624] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15619.849286] LNet: Added LNI 192.168.201.106@tcp32 [8/256/0/180] [15626.822331] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15626.868626] LNet: Added LNI 192.168.201.106@tcp33 [8/256/0/180] [15633.775209] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15633.831170] LNet: Added LNI 192.168.201.106@tcp34 [8/256/0/180] [15641.632570] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15641.670205] LNet: Added LNI 192.168.201.106@tcp35 [8/256/0/180] [15648.924517] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15648.955904] LNet: Added LNI 192.168.201.106@tcp36 [8/256/0/180] [15656.681447] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15656.711718] LNet: Added LNI 192.168.201.106@tcp37 [8/256/0/180] [15664.788480] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15664.818768] LNet: Added LNI 192.168.201.106@tcp38 [8/256/0/180] [15673.187572] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15673.207755] LNet: Added LNI 192.168.201.106@tcp39 [8/256/0/180] [15681.074382] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15681.096425] LNet: Added LNI 192.168.201.106@tcp40 [8/256/0/180] [15688.824534] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15688.855995] LNet: Added LNI 192.168.201.106@tcp41 [8/256/0/180] [15697.249776] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15697.274091] LNet: Added LNI 192.168.201.106@tcp42 [8/256/0/180] [15706.423582] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15706.462846] LNet: Added LNI 192.168.201.106@tcp43 [8/256/0/180] [15716.514567] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15716.591225] LNet: Added LNI 192.168.201.106@tcp44 [8/256/0/180] [15729.343989] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15729.387216] LNet: Added LNI 192.168.201.106@tcp45 [8/256/0/180]