[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 469103360 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002011] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011140] LSM: Security Framework initializing [ 0.012072] Yama: becoming mindful. [ 0.014054] SELinux: Initializing. [ 0.015340] *** VALIDATE selinux *** [ 0.030226] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.038922] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.040157] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.041115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.043011] *** VALIDATE tmpfs *** [ 0.045655] *** VALIDATE proc *** [ 0.047089] *** VALIDATE cgroup *** [ 0.048013] *** VALIDATE cgroup2 *** [ 0.049475] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.051003] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.052012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.053031] Spectre V2 : User space: Vulnerable [ 0.054014] Speculative Store Bypass: Vulnerable [ 0.057222] debug: unmapping init [mem 0xffffffff88c59000-0xffffffff88c60fff] [ 0.060000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.060795] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.062028] ... version: 2 [ 0.063019] ... bit width: 48 [ 0.064019] ... generic registers: 4 [ 0.065018] ... value mask: 0000ffffffffffff [ 0.066016] ... max period: 00007fffffffffff [ 0.067018] ... fixed-purpose events: 3 [ 0.068015] ... event mask: 000000070000000f [ 0.069319] rcu: Hierarchical SRCU implementation. [ 0.074161] smp: Bringing up secondary CPUs ... [ 0.076035] x86: Booting SMP configuration: [ 0.077016] .... node #0, CPUs: #1 #2 #3 [ 0.086168] smp: Brought up 1 node, 4 CPUs [ 0.088017] smpboot: Max logical packages: 1 [ 0.089196] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.128553] node 0 deferred pages initialised in 37ms [ 0.133215] devtmpfs: initialized [ 0.134303] x86/mm: Memory block size: 128MB [ 0.137938] gcov: version magic: 0x41383552 [ 0.139375] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.140248] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.141348] pinctrl core: initialized pinctrl subsystem [ 0.142228] [ 0.143015] ************************************************************* [ 0.144015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.145079] ** ** [ 0.146014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.147013] ** ** [ 0.148015] ** This means that this kernel is built to expose internal ** [ 0.149014] ** IOMMU data structures, which may compromise security on ** [ 0.150012] ** your system. ** [ 0.151010] ** ** [ 0.152009] ** If you see this message and you are not debugging the ** [ 0.153013] ** kernel, report this immediately to your vendor! ** [ 0.154017] ** ** [ 0.155015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156015] ************************************************************* [ 0.157908] NET: Registered protocol family 16 [ 0.158499] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.159055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.160175] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.161749] cpuidle: using governor menu [ 0.164086] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.166623] PCI: Using configuration type 1 for base access [ 0.169289] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.177117] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.178000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.178428] cryptd: max_cpu_qlen set to 1000 [ 0.180217] ACPI: Added _OSI(Module Device) [ 0.181000] ACPI: Added _OSI(Processor Device) [ 0.181000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.184013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.190009] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.197145] ACPI: Interpreter enabled [ 0.198058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.199012] ACPI: Using IOAPIC for interrupt routing [ 0.200103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.202000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.215034] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.218078] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.221021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.228083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.236083] acpiphp: Slot [2] registered [ 0.238179] acpiphp: Slot [5] registered [ 0.240198] acpiphp: Slot [6] registered [ 0.242181] acpiphp: Slot [7] registered [ 0.244226] acpiphp: Slot [8] registered [ 0.246183] acpiphp: Slot [9] registered [ 0.248107] acpiphp: Slot [10] registered [ 0.250121] acpiphp: Slot [3] registered [ 0.252119] acpiphp: Slot [4] registered [ 0.253126] acpiphp: Slot [11] registered [ 0.255162] acpiphp: Slot [12] registered [ 0.257161] acpiphp: Slot [13] registered [ 0.260124] acpiphp: Slot [14] registered [ 0.261114] acpiphp: Slot [15] registered [ 0.265262] acpiphp: Slot [16] registered [ 0.266160] acpiphp: Slot [17] registered [ 0.268178] acpiphp: Slot [18] registered [ 0.270144] acpiphp: Slot [19] registered [ 0.271088] acpiphp: Slot [20] registered [ 0.273114] acpiphp: Slot [21] registered [ 0.274070] acpiphp: Slot [22] registered [ 0.275484] acpiphp: Slot [23] registered [ 0.281322] acpiphp: Slot [24] registered [ 0.283131] acpiphp: Slot [25] registered [ 0.285086] acpiphp: Slot [26] registered [ 0.287106] acpiphp: Slot [27] registered [ 0.290718] acpiphp: Slot [28] registered [ 0.291137] acpiphp: Slot [29] registered [ 0.294098] acpiphp: Slot [30] registered [ 0.298130] acpiphp: Slot [31] registered [ 0.299096] PCI host bridge to bus 0000:00 [ 0.304025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.309041] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.311025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.314022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.317018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.323029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.326319] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.329044] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.333606] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.346604] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.352062] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.357027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.359018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.362015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.365889] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.369000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.370099] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.374695] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.381016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.397020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.409016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.416431] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.424017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.438018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.464019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.476612] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.489018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.495014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.508017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.515933] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.519000] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.526017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.542015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.553061] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.562016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.569016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.597013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.609619] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.623016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.630015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.645020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.658070] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.664015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.669015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.678047] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.688600] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.691503] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.696029] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.701012] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.706237] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.713032] iommu: Default domain type: Passthrough [ 0.714817] SCSI subsystem initialized [ 0.715134] ACPI: bus type USB registered [ 0.716000] usbcore: registered new interface driver usbfs [ 0.717082] usbcore: registered new interface driver hub [ 0.718208] usbcore: registered new device driver usb [ 0.720272] pps_core: LinuxPPS API ver. 1 registered [ 0.722010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.725065] PTP clock support registered [ 0.726497] EDAC MC: Ver: 3.0.0 [ 0.727872] PCI: Using ACPI for IRQ routing [ 0.730086] NetLabel: Initializing [ 0.731007] NetLabel: domain hash size = 128 [ 0.733011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.739086] NetLabel: unlabeled traffic allowed by default [ 0.745459] vgaarb: loaded [ 0.748114] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.754013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.767491] clocksource: Switched to clocksource kvm-clock [ 1.079237] VFS: Disk quotas dquot_6.6.0 [ 1.082852] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.091957] *** VALIDATE ramfs *** [ 1.096392] *** VALIDATE hugetlbfs *** [ 1.101812] pnp: PnP ACPI init [ 1.105453] pnp: PnP ACPI: found 6 devices [ 1.155411] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.160080] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.163134] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.165807] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.168689] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.173264] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.176805] NET: Registered protocol family 2 [ 1.179551] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.202237] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.214883] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.228904] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.239587] TCP: Hash tables configured (established 65536 bind 65536) [ 1.246049] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.249942] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.254131] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.261848] NET: Registered protocol family 1 [ 1.269616] RPC: Registered named UNIX socket transport module. [ 1.274864] RPC: Registered udp transport module. [ 1.276527] RPC: Registered tcp transport module. [ 1.278572] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.281885] NET: Registered protocol family 44 [ 1.284095] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.287698] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.290399] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.293908] PCI: CLS 0 bytes, default 64 [ 1.296092] Unpacking initramfs... [ 4.579889] debug: unmapping init [mem 0xffff9973fcc54000-0xffff9973fffbffff] [ 4.585101] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.588270] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.592306] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 7.041426] Initialise system trusted keyrings [ 7.046530] Key type blacklist registered [ 7.049827] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 7.075783] zbud: loaded [ 7.093814] *** VALIDATE nfs *** [ 7.096217] *** VALIDATE nfs4 *** [ 7.107275] pstore: using deflate compression [ 7.128475] Platform Keyring initialized [ 7.605957] NET: Registered protocol family 38 [ 7.609459] Key type asymmetric registered [ 7.610492] Asymmetric key parser 'x509' registered [ 7.614686] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 7.625888] io scheduler mq-deadline registered [ 7.632465] io scheduler kyber registered [ 7.637156] io scheduler bfq registered [ 7.651814] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.664792] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.680744] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.685623] ACPI: Power Button [PWRF] [ 7.710361] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.723403] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.762867] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.776647] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.811072] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.842565] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.875482] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.882611] Non-volatile memory driver v1.3 [ 7.884848] Linux agpgart interface v0.103 [ 7.938345] virtio_blk virtio1: [vda] 134472 512-byte logical blocks (68.8 MB/65.7 MiB) [ 7.944936] vda: detected capacity change from 0 to 68849664 [ 7.973499] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.977726] vdb: detected capacity change from 0 to 1073741824 [ 7.998806] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.002821] vdc: detected capacity change from 0 to 2621440000 [ 8.023700] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.028350] vdd: detected capacity change from 0 to 2621440000 [ 8.051883] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.055962] vde: detected capacity change from 0 to 4294967296 [ 8.074516] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.078398] vdf: detected capacity change from 0 to 4294967296 [ 8.095707] libphy: Fixed MDIO Bus: probed [ 8.122058] usbcore: registered new interface driver usbserial_generic [ 8.124844] usbserial: USB Serial support registered for generic [ 8.127646] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.132977] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.135039] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.139169] mousedev: PS/2 mouse device common for all mice [ 8.146329] rtc_cmos 00:05: RTC can wake from S4 [ 8.151904] rtc_cmos 00:05: registered as rtc0 [ 8.152655] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.155043] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.155088] intel_pstate: CPU model not supported [ 8.174369] hid: raw HID events driver (C) Jiri Kosina [ 8.175771] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.178831] usbcore: registered new interface driver usbhid [ 8.178842] usbhid: USB HID core driver [ 8.180388] drop_monitor: Initializing network drop monitor service [ 8.195588] Initializing XFRM netlink socket [ 8.199972] NET: Registered protocol family 10 [ 8.203154] Segment Routing with IPv6 [ 8.205052] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.205560] NET: Registered protocol family 17 [ 8.207096] mpls_gso: MPLS GSO support [ 8.221487] RAS: Correctable Errors collector initialized. [ 8.224870] AVX version of gcm_enc/dec engaged. [ 8.227319] AES CTR mode by8 optimization enabled [ 8.337866] sched_clock: Marking stable (8337791676, 0)->(9768947683, -1431156007) [ 8.342608] registered taskstats version 1 [ 8.345395] Loading compiled-in X.509 certificates [ 8.348484] zswap: loaded using pool lzo/zbud [ 8.382246] Key type big_key registered [ 8.398824] Key type encrypted registered [ 8.400615] ima: No TPM chip found, activating TPM-bypass! [ 8.402770] ima: Allocated hash algorithm: sha1 [ 8.404310] ima: No architecture policies found [ 8.405893] evm: Initialising EVM extended attributes: [ 8.407667] evm: security.selinux [ 8.408575] evm: security.ima [ 8.409399] evm: security.capability [ 8.410272] evm: HMAC attrs: 0x1 [ 8.412418] rtc_cmos 00:05: setting system clock to 2026-09-05 09:39:18 UTC (1788601158) [ 8.421337] debug: unmapping init [mem 0xffffffff89c03000-0xffffffff89dfffff] [ 8.426326] debug: unmapping init [mem 0xffffffff88982000-0xffffffff88c58fff] [ 8.436112] Write protecting the kernel read-only data: 28672k [ 8.440143] debug: unmapping init [mem 0xffffffff87003000-0xffffffff871fffff] [ 8.456023] debug: unmapping init [mem 0xffffffff87914000-0xffffffff879fffff] [ 8.581881] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.597659] systemd[1]: Detected virtualization kvm. [ 8.602393] systemd[1]: Detected architecture x86-64. [ 8.604872] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.642528] systemd[1]: No hostname configured. [ 8.645662] systemd[1]: Set hostname to . [ 8.649936] random: systemd: uninitialized urandom read (16 bytes read) [ 8.653382] systemd[1]: Initializing machine ID from random generator. [ 9.052745] random: systemd: uninitialized urandom read (16 bytes read) [ 9.056478] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 9.066770] random: systemd: uninitialized urandom read (16 bytes read) [ 9.070705] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 9.075171] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.912436] device-mapper: uevent: version 1.0.3 [ 10.915788] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 13.444579] virtio_net virtio0 ens2: renamed from eth0 [ 13.530556] random: fast init done [ 13.739525] scsi host0: ata_piix [ 13.911661] scsi host1: ata_piix [ 13.915507] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 13.921975] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.924410] random: crng init done [ 18.935261] random: 7 urandom warning(s) missed due to ratelimiting [ 20.879787] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 23.413525] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 27.145557] printk: systemd: 26 output lines suppressed due to ratelimiting [ 28.033922] SELinux: Disabled at runtime. [ 28.160365] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 28.180975] systemd[1]: Detected virtualization kvm. [ 28.189733] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 30.363215] systemd[1]: initrd-switch-root.service: Succeeded. [ 30.369810] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 30.386323] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 30.402724] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 30.409691] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 30.428293] systemd[1]: Starting Journal Service... Starting Journal Service... [ 30.455572] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ 30.653064] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 32.060530] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 33.415379] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 33.479413] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 34.698481] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 34.802579] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 39.706141] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 40.340121] NFS: Registering the id_resolver key type [ 40.341710] Key type id_resolver registered [ 40.342852] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg103-server login: [ 108.676489] libcfs: loading out-of-tree module taints kernel. [ 108.747207] Key type ._llcrypt registered [ 108.749919] Key type .llcrypt registered [ 108.950351] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 125.356977] hrtimer: interrupt took 6107217 ns [ 129.608738] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 131.812715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 131.856642] alg: No test for adler32 (adler32-zlib) [ 133.461532] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 134.698677] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 136.505236] Key type lgssc registered [ 138.615490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 157.240977] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 198.614748] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 214.746252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 214.818097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.324676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 216.350388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 216.496901] Lustre: lustre-MDT0000: new disk, initializing [ 216.686679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 216.711689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 222.108806] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.194224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 237.289060] Lustre: 6509:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 237.331533] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 237.334599] Lustre: Skipped 1 previous similar message [ 237.387062] Lustre: lustre-MDT0001: new disk, initializing [ 237.429778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 237.443738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 237.451043] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 241.710768] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 246.538872] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 256.312352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 256.693916] Lustre: lustre-OST0000: new disk, initializing [ 256.697515] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 256.814068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 259.871123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 259.883886] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 259.961730] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 262.948082] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 277.264727] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 277.387803] Lustre: lustre-OST0001: new disk, initializing [ 277.393334] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 277.454584] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 283.238148] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 287.253868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 287.270974] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 287.333963] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 294.944790] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 299.148697] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 310.287547] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing check_logdir /tmp/testlogs/ [ 314.007245] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing yml_node [ 317.845319] Lustre: DEBUG MARKER: Client: 2.17.51.74 [ 320.906712] Lustre: DEBUG MARKER: MDS: 2.17.51.74 [ 323.723734] Lustre: DEBUG MARKER: OSS: 2.17.51.74 [ 325.375582] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 05:44:34 EDT 2026 [ 347.221178] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 349.391778] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 365.539097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 365.548610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.579629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 369.128245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 369.142822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 369.147201] Lustre: Skipped 1 previous similar message [ 369.170124] Lustre: Skipped 3 previous similar messages [ 371.488341] Lustre: server umount lustre-MDT0000 complete [ 375.221684] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788601525 with bad export cookie 4474584845003660739 [ 375.226596] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 375.231074] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 379.360330] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 379.364096] LustreError: 8857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 379.373870] Lustre: Skipped 1 previous similar message [ 379.386642] LustreError: 8857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 379.394528] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 380.384321] LustreError: 10027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 380.401979] LustreError: 10027:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 381.595528] Lustre: server umount lustre-MDT0001 complete [ 385.587038] Lustre: server umount lustre-OST0000 complete [ 389.287484] Lustre: server umount lustre-OST0001 complete [ 395.746575] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 404.488732] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 438.544115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 438.956932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 438.991447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 439.050249] Lustre: lustre-MDT0000: new disk, initializing [ 439.149981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 439.191251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 444.318930] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 456.251256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 456.436868] Lustre: 14988:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 456.563797] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 456.567591] Lustre: Skipped 1 previous similar message [ 456.677955] Lustre: lustre-MDT0001: new disk, initializing [ 456.781251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 456.825124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 456.840060] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 461.933749] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 470.698250] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 474.505612] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 482.921017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 483.359319] Lustre: lustre-OST0000: new disk, initializing [ 483.369389] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 483.537443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 484.504648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 484.516750] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 484.684507] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 490.005538] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 497.884230] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 502.755137] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 502.997198] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 507.294323] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 507.524380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 510.432073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 510.443719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.446758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 510.457659] Lustre: Skipped 2 previous similar messages [ 510.470328] Lustre: Skipped 4 previous similar messages [ 515.491418] Lustre: server umount lustre-OST0000 complete [ 520.675755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 520.690543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 520.696832] Lustre: Skipped 1 previous similar message [ 520.710934] Lustre: Skipped 1 previous similar message [ 525.699791] Lustre: server umount lustre-MDT0000 complete [ 525.794745] LustreError: 14995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.801733] LustreError: 14995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 529.437890] LustreError: 14981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788601679 with bad export cookie 4474584845003662272 [ 529.445638] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 529.456819] LustreError: 14981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 529.766508] Lustre: server umount lustre-MDT0001 complete [ 533.624700] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 05:48:02 (1788601682) [ 536.329688] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 538.366695] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 05:48:07 (1788601687) [ 545.582301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 546.414672] LustreError: 18996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.568250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 551.324739] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 551.917498] LustreError: 18997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.215883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.939522] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 570.367594] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 573.726811] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 582.285544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 589.209586] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 596.197675] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 606.689047] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 606.697268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 606.706333] LustreError: Skipped 2 previous similar messages [ 606.707839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 606.718720] Lustre: Skipped 1 previous similar message [ 606.739931] Lustre: Skipped 1 previous similar message [ 610.416802] Lustre: server umount lustre-OST0000 complete [ 616.929356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 620.425361] Lustre: server umount lustre-MDT0000 complete [ 622.060848] LustreError: 21237:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 622.079063] LustreError: 21237:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 624.057239] LustreError: 18978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788601774 with bad export cookie 4474584845003663161 [ 624.060378] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 624.070669] LustreError: 18978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 624.363335] Lustre: server umount lustre-MDT0001 complete [ 634.394957] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 637.290545] Key type lgssc unregistered [ 637.579747] LNet: 22175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 637.583369] LNetError: 22175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 637.598219] LNet: Removed LNI 192.168.201.103@tcp [ 638.667132] Key type .llcrypt unregistered [ 638.671199] Key type ._llcrypt unregistered [ 662.561918] Key type ._llcrypt registered [ 662.566186] Key type .llcrypt registered [ 662.690688] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 664.020892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 664.140404] alg: No test for adler32 (adler32-zlib) [ 665.307069] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 665.689533] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 667.375564] Key type lgssc registered [ 668.586298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 677.892946] loop: module loaded [ 678.004738] loop0: detected capacity change from 0 to 204800000 [ 678.268651] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 678.304798] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 678.624592] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 685.405960] loop0: detected capacity change from 0 to 204800000 [ 685.709232] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 685.721485] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 685.745770] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 691.603655] loop0: detected capacity change from 0 to 204800000 [ 697.575868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 697.597429] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.892645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 698.962974] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 699.065783] Lustre: lustre-MDT0000: new disk, initializing [ 699.143672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 699.164659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 703.203563] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 711.127354] loop1: detected capacity change from 0 to 204800000 [ 717.306967] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 717.597172] Lustre: lustre-OST07c6: new disk, initializing [ 717.604155] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 717.615366] Lustre: Skipped 1 previous similar message [ 717.665686] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 723.500926] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 726.162860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 726.187402] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 726.238818] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 728.072292] Lustre: 24335:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 728.082549] Lustre: 24335:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 728.091241] Lustre: 24335:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 728.097300] Lustre: 24335:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 728.105103] Lustre: 24335:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 728.113318] Lustre: 24335:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 745.723380] Lustre: 24335:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 745.732861] Lustre: 24335:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 745.741833] Lustre: 24335:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 745.753848] Lustre: 24335:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 745.761328] Lustre: 24335:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 745.767330] Lustre: 24335:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 750.869531] Lustre: 25980:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 750.877272] Lustre: 25980:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 750.888160] Lustre: 25980:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 750.899773] Lustre: 25980:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 750.908160] Lustre: 25980:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 750.918975] Lustre: 25980:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 750.925912] Lustre: 25980:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 750.935385] Lustre: 25980:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 750.941103] Lustre: 25980:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 750.946802] Lustre: 25980:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 750.957107] Lustre: 25980:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 750.961081] Lustre: 25980:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.190829] Lustre: 24334:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 754.209378] Lustre: 24334:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 754.214411] Lustre: 24334:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 754.222520] Lustre: 24334:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.233847] Lustre: 24334:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 754.246070] Lustre: 24334:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.257721] Lustre: 24334:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 754.270104] Lustre: 24334:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.280074] Lustre: 24334:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 754.293751] Lustre: 24334:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.300288] Lustre: 24334:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 754.307175] Lustre: 24334:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 759.411142] Lustre: 24334:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 759.433650] Lustre: 24334:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 759.448563] Lustre: 24334:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 759.462621] Lustre: 24334:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 759.471773] Lustre: 24334:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 759.479938] Lustre: 24334:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 759.488049] Lustre: 24334:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 759.494752] Lustre: 24334:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 759.500959] Lustre: 24334:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 759.507266] Lustre: 24334:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 759.516049] Lustre: 24334:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 759.525875] Lustre: 24334:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.321110] Lustre: 25980:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 770.339609] Lustre: 25980:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 5 previous similar messages [ 770.353643] Lustre: 25980:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 770.362848] Lustre: 25980:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 770.367564] Lustre: 25980:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 770.376187] Lustre: 25980:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 770.391323] Lustre: 25980:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 770.405625] Lustre: 25980:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 770.417393] Lustre: 25980:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 770.421900] Lustre: 25980:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 770.431450] Lustre: 25980:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 770.436633] Lustre: 25980:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 818.155412] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 818.167436] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 818.206682] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 823.336338] Lustre: server umount lustre-OST07c6 complete [ 833.226092] Lustre: server umount lustre-MDT0000 complete [ 851.498199] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 854.850641] Key type lgssc unregistered [ 855.209169] LNet: 27860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.218893] LNetError: 27860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.293911] LNet: Removed LNI 192.168.201.103@tcp [ 857.457221] Key type .llcrypt unregistered [ 857.459330] Key type ._llcrypt unregistered [ 880.655941] Key type ._llcrypt registered [ 880.657696] Key type .llcrypt registered [ 880.806531] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 882.497586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 882.567108] alg: No test for adler32 (adler32-zlib) [ 883.629676] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 883.944770] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 885.687457] Key type lgssc registered [ 887.118635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 895.141024] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 895.171837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 896.690256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 901.790993] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 906.208913] LustreError: 29112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 906.237713] LustreError: 29112:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 911.336737] LustreError: 29111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 912.144852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 912.718305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 917.343062] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 923.098485] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 926.888670] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 934.621084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 935.065023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 941.257066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 942.139583] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 948.197700] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 967.141694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 967.141989] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 967.156459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 967.172320] Lustre: Skipped 1 previous similar message [ 972.256674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 972.270976] Lustre: Skipped 1 previous similar message [ 977.383620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 977.407146] Lustre: Skipped 1 previous similar message [ 978.401084] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 978.674666] Lustre: server umount lustre-OST0000 complete [ 982.509294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 982.532567] Lustre: Skipped 1 previous similar message [ 982.546334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 982.553490] Lustre: Skipped 1 previous similar message [ 987.620985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 987.629160] Lustre: Skipped 1 previous similar message [ 988.953413] Lustre: server umount lustre-MDT0000 complete [ 992.742024] LustreError: 29106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 992.758535] LustreError: 29106:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 993.158555] LustreError: 29095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788602143 with bad export cookie 3780321064461115795 [ 993.173364] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 993.559395] Lustre: server umount lustre-MDT0001 complete [ 1005.231765] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1008.190865] Key type lgssc unregistered [ 1008.510611] LNet: 32497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1008.518420] LNetError: 32497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1008.542945] LNet: Removed LNI 192.168.201.103@tcp [ 1009.391749] Key type .llcrypt unregistered [ 1009.393773] Key type ._llcrypt unregistered [ 1032.853450] Key type ._llcrypt registered [ 1032.856210] Key type .llcrypt registered [ 1032.989845] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1034.324303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1034.350101] alg: No test for adler32 (adler32-zlib) [ 1035.418745] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 1035.619357] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1037.287379] Key type lgssc registered [ 1038.493147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1047.165505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1047.201496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1048.927898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1052.954727] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1054.181063] LustreError: 33756:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1054.214147] LustreError: 33756:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1058.277291] LustreError: 33757:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1063.402220] LustreError: 33756:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1065.117972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1065.643719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1071.347264] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1076.871475] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1081.659913] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1088.261858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1088.582509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1093.719771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1095.223616] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1101.951694] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1119.212360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1119.217516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1119.232839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1119.287877] Lustre: Skipped 1 previous similar message [ 1123.123110] Lustre: server umount lustre-OST0000 complete [ 1129.442558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1129.471220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1129.481142] Lustre: Skipped 2 previous similar messages [ 1134.565889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1134.659633] Lustre: server umount lustre-MDT0000 complete [ 1138.559617] LustreError: 33737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788602288 with bad export cookie 7617012479001857864 [ 1138.562141] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1138.564862] LustreError: 33737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1138.817602] Lustre: server umount lustre-MDT0001 complete [ 1149.801877] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1152.857798] Key type lgssc unregistered [ 1153.251930] LNet: 36961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1153.254806] LNetError: 36961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1153.268626] LNet: Removed LNI 192.168.201.103@tcp [ 1154.071037] Key type .llcrypt unregistered [ 1154.075766] Key type ._llcrypt unregistered [ 1177.771652] Key type ._llcrypt registered [ 1177.776567] Key type .llcrypt registered [ 1177.907174] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1178.932698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1179.044251] alg: No test for adler32 (adler32-zlib) [ 1180.363288] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 1180.597778] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1182.255498] Key type lgssc registered [ 1183.654116] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1192.204749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1192.228767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1193.718188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1197.778499] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1203.170637] LustreError: 38359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1203.191434] LustreError: 38359:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1208.115364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1208.298475] LustreError: 38360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1208.630076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1213.183720] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1218.982448] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1223.144700] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1231.391788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1231.819791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1238.013818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1240.441266] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1247.344994] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1258.467152] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1258.468211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1258.478501] Lustre: Skipped 1 previous similar message [ 1258.505736] Lustre: Skipped 2 previous similar messages [ 1260.794625] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 1262.382490] Lustre: server umount lustre-MDT0000 complete [ 1263.594989] LustreError: 38356:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1263.650686] LustreError: 38356:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1265.865572] LustreError: 40514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1266.718662] LustreError: 38340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788602416 with bad export cookie 13221546573975646189 [ 1266.760699] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1268.712877] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.725581] Lustre: Skipped 1 previous similar message [ 1268.734147] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1270.982272] LustreError: 40220:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1271.014953] LustreError: 40220:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1271.020084] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.3@tcp (stopping) [ 1273.158554] Lustre: server umount lustre-MDT0001 complete [ 1295.585678] Lustre: server umount lustre-OST0000 complete [ 1316.193994] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1318.828481] Key type lgssc unregistered [ 1319.050497] LNet: 41880:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1319.063199] LNetError: 41880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1319.075170] LNet: Removed LNI 192.168.201.103@tcp [ 1319.965445] Key type .llcrypt unregistered [ 1319.967857] Key type ._llcrypt unregistered [ 1344.014168] Key type ._llcrypt registered [ 1344.015995] Key type .llcrypt registered [ 1344.115305] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1345.069110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1345.119575] alg: No test for adler32 (adler32-zlib) [ 1346.282370] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 1346.474630] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1348.191256] Key type lgssc registered [ 1349.535082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1358.373599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1358.399466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1359.833557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1364.660174] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1369.062116] LustreError: 43282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1369.094045] LustreError: 43282:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1374.176943] LustreError: 43281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1374.230111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1374.674730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1379.306259] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1385.270065] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1389.186088] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1396.307458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1396.746592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1404.475154] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1405.990575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1411.636656] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1418.454235] Lustre: lustre-OST0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 1421.286290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1421.294745] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1421.315922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1421.342090] Lustre: Skipped 1 previous similar message [ 1423.565830] LustreError: 44877:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1423.925859] Lustre: server umount lustre-OST0000 complete [ 1441.094676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1441.111238] Lustre: Skipped 1 previous similar message [ 1441.120287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1447.394544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1447.405721] Lustre: Skipped 4 previous similar messages [ 1447.525466] Lustre: server umount lustre-MDT0000 complete [ 1451.216452] LustreError: 43263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788602601 with bad export cookie 8700331696608008843 [ 1451.246099] LustreError: 43263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1451.246474] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1457.565135] Lustre: server umount lustre-MDT0001 complete [ 1473.103660] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1477.220820] Key type lgssc unregistered [ 1477.733838] LNet: 46638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1477.742487] LNetError: 46638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1477.772276] LNet: Removed LNI 192.168.201.103@tcp [ 1479.007908] Key type .llcrypt unregistered [ 1479.015957] Key type ._llcrypt unregistered [ 1507.897530] Key type ._llcrypt registered [ 1507.899992] Key type .llcrypt registered [ 1508.078901] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1509.310891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1509.571061] alg: No test for adler32 (adler32-zlib) [ 1511.028704] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 1511.350428] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1513.007308] Key type lgssc registered [ 1514.206878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1521.985972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1522.021059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1523.638092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1528.861300] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1533.936888] LustreError: 48038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1533.961895] LustreError: 48038:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1538.720316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1539.051928] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1539.202050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1543.819721] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1549.463345] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1553.531852] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1560.974754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1561.456960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1568.264816] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1569.665698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1575.508297] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1581.246216] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1584.818860] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1596.968852] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-05 6h05m45s [ 1598.400436] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1602.792136] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 1603.552862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1603.567808] Lustre: Skipped 2 previous similar messages [ 1603.846949] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 1603.854529] Lustre: Skipped 3 previous similar messages [ 1606.901967] Lustre: server umount lustre-MDT0000 complete [ 1608.906827] LustreError: 49338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1610.583401] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1610.736750] LustreError: 49027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1610.763337] LustreError: 49027:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1615.047737] LustreError: 48033:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1617.904058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1618.093298] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1618.341900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1618.415737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1619.383051] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1623.384489] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1627.208167] Lustre: DEBUG MARKER: Wait for df (36111) ... [ 1628.854023] Lustre: DEBUG MARKER: done [ 1634.790576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1634.792524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1634.811348] Lustre: Skipped 1 previous similar message [ 1634.828974] Lustre: Skipped 4 previous similar messages [ 1639.908365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1639.923748] Lustre: Skipped 1 previous similar message [ 1640.168984] Lustre: server umount lustre-OST0000 complete [ 1650.148367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1650.157464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1650.165616] Lustre: Skipped 1 previous similar message [ 1650.209258] Lustre: Skipped 1 previous similar message [ 1659.359755] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1659.593989] Lustre: server umount lustre-MDT0000 complete [ 1663.474239] LustreError: 50196:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1663.510723] LustreError: 50196:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1664.562310] LustreError: 48019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788602814 with bad export cookie 14078276423452913315 [ 1664.584807] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1665.177594] Lustre: server umount lustre-MDT0001 complete [ 1677.945039] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1681.093459] Key type lgssc unregistered [ 1681.502695] LNet: 52600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1681.516878] LNetError: 52600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1682.534168] LNet: Removed LNI 192.168.201.103@tcp [ 1683.282534] Key type .llcrypt unregistered [ 1683.284643] Key type ._llcrypt unregistered [ 1721.827537] Key type ._llcrypt registered [ 1721.829325] Key type .llcrypt registered [ 1721.982496] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1723.560258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1723.604925] alg: No test for adler32 (adler32-zlib) [ 1724.772146] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 1725.035566] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1726.711170] Key type lgssc registered [ 1728.065399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1736.582590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1736.621642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1738.013519] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1738.041290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1738.301062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1743.331926] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1753.866266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1754.006572] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1754.031519] Lustre: 54792:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1754.328733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1759.120831] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1764.563897] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1767.948808] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1775.339343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1775.511234] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1775.824610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1782.419871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1782.719579] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1788.907914] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1795.720741] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-05 6h09m04s [ 1797.193409] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1804.829612] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1807.840880] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1807.942133] LustreError: 56981:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1828.113367] Lustre: lustre-MDT0000: Client d9fdcb57-f27e-4c80-9051-133738bd8ed7 (at 192.168.201.3@tcp) reconnecting [ 1848.023208] LustreError: 56981:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1851.178825] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1854.304501] Lustre: DEBUG MARKER: done [ 1864.162348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1864.166661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1864.178497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1864.698286] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1864.708812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1864.731771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1866.644326] Lustre: server umount lustre-OST0000 complete [ 1872.352479] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1872.374287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1872.401879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1875.489114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1875.499665] Lustre: Skipped 4 previous similar messages [ 1876.540344] Lustre: server umount lustre-MDT0000 complete [ 1880.754695] LustreError: 54785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788603030 with bad export cookie 682030654352407826 [ 1880.760310] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1880.770730] LustreError: 54785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1881.048470] Lustre: server umount lustre-MDT0001 complete [ 1892.697688] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1896.205849] Key type lgssc unregistered [ 1896.595893] LNet: 58764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1896.609454] LNetError: 58764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1896.636209] LNet: Removed LNI 192.168.201.103@tcp [ 1897.425424] Key type .llcrypt unregistered [ 1897.429364] Key type ._llcrypt unregistered [ 1938.676954] Key type ._llcrypt registered [ 1938.680259] Key type .llcrypt registered [ 1938.829540] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1940.466074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1940.503452] alg: No test for adler32 (adler32-zlib) [ 1941.567218] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 1941.825776] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1943.559238] Key type lgssc registered [ 1944.524234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1947.467688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1947.484415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1948.775348] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1948.792224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1949.004840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1950.131873] Lustre: Failing over lustre-MDT0000 [ 1950.575657] Lustre: server umount lustre-MDT0000 complete [ 1957.471774] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 06:11:46 (1788603106) [ 1964.561394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1965.106670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1970.482200] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1980.224114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1980.469489] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1980.585163] Lustre: 61548:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1981.064667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1987.480828] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1993.786306] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1997.408261] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2004.096776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2004.353587] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2004.536477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2010.866650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2010.932258] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2016.203703] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2022.396948] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2031.589646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2031.591657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2031.622111] Lustre: Skipped 1 previous similar message [ 2031.630245] Lustre: Skipped 1 previous similar message [ 2035.694109] Lustre: server umount lustre-MDT0000 complete [ 2039.187358] LustreError: 63179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788603189 with bad export cookie 13805185084070185477 [ 2039.194208] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2039.200223] LustreError: 63179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2039.572349] Lustre: server umount lustre-MDT0001 complete [ 2044.764195] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2050.895375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2056.927169] Lustre: 60473:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788603191/real 1788603191] req@00000000c640f36e x1875486274788864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788603207 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2056.940663] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2056.948378] Lustre: Skipped 1 previous similar message [ 2059.997850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2060.255348] Lustre: 60474:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788603194/real 1788603194] req@00000000fb3f8cb7 x1875486274788992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788603210 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2062.303633] Lustre: 60474:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788603196/real 1788603196] req@0000000065f2c84c x1875486274789120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788603212 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2064.539048] LustreError: 63171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2064.554423] LustreError: 63171:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2064.676559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2064.795150] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2065.375310] Lustre: 60472:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788603199/real 1788603199] req@000000007dabe4c9 x1875486274789376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788603215 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2066.723657] LustreError: 63172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2067.748053] LustreError: 64978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2067.773748] LustreError: 64978:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2069.216741] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2070.815107] Lustre: 60472:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788603204/real 1788603204] req@00000000e84040f8 x1875486274789504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788603220 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2073.057273] LustreError: 64980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2073.073126] LustreError: 64980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2077.822744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2078.055637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2081.901630] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2083.330535] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2086.454968] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2086.612098] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2090.537938] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2096.616718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2096.634065] LustreError: 66118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2096.653359] LustreError: 66118:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2096.772326] Lustre: server umount lustre-MDT0000 complete [ 2100.133116] LustreError: 61542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788603250 with bad export cookie 13805185084070187752 [ 2100.135273] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2100.140390] LustreError: 61542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2100.415317] Lustre: server umount lustre-MDT0001 complete [ 2106.575659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2115.464470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2117.856436] Lustre: 60473:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788603251/real 1788603251] req@000000008900a045 x1875486274813568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788603267 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2117.892421] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2117.902341] Lustre: Skipped 2 previous similar messages [ 2125.087824] LustreError: 60471:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a9b3c1bb x1875486274814592/t0(0) o250->MGC192.168.201.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2125.695880] LustreError: 63171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2125.728557] LustreError: 63171:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2125.905883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2125.991612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2129.813844] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2136.043581] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2138.535664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2143.845636] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2144.246830] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2149.249204] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2152.428919] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2159.589171] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2159.590574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2159.609820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2159.609828] Lustre: Skipped 1 previous similar message [ 2159.787659] Lustre: server umount lustre-MDT0000 complete [ 2163.101683] LustreError: 61541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788603313 with bad export cookie 13805185084070189131 [ 2163.103098] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2163.110601] LustreError: 61541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2163.377604] Lustre: server umount lustre-MDT0001 complete [ 2169.649777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2173.101905] Lustre: DEBUG MARKER: files compared the same [ 2184.984803] Lustre: server umount lustre-OST0000 complete [ 2201.540352] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2204.565227] Key type lgssc unregistered [ 2204.864939] LNet: 70469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2204.875022] LNetError: 70469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2204.892775] LNet: Removed LNI 192.168.201.103@tcp [ 2205.739614] Key type .llcrypt unregistered [ 2205.742116] Key type ._llcrypt unregistered [ 2229.359371] Key type ._llcrypt registered [ 2229.364184] Key type .llcrypt registered [ 2229.474095] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2230.279380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2230.331681] alg: No test for adler32 (adler32-zlib) [ 2231.407716] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 2231.597403] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2233.247793] Key type lgssc registered [ 2234.393818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2240.803744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2240.816994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2242.351851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2246.427860] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug +malloc all [ 2252.779427] LustreError: 71869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2252.791902] LustreError: 71869:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2254.146426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2254.409759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2258.905857] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug +malloc all [ 2264.551000] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2267.786791] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2274.138603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2274.464648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2280.960223] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug +malloc all [ 2286.409500] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2287.806949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2298.338587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2298.342584] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2298.363841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2302.618826] Lustre: server umount lustre-OST0000 complete [ 2308.576111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2308.601833] Lustre: Skipped 1 previous similar message [ 2308.609109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2308.611649] Lustre: Skipped 1 previous similar message [ 2313.118562] Lustre: server umount lustre-MDT0000 complete [ 2313.696733] LustreError: 71865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2313.722526] LustreError: 71865:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2318.194941] LustreError: 72588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788603468 with bad export cookie 1772907922676080544 [ 2318.207704] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2318.208550] LustreError: 72588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2318.691229] Lustre: server umount lustre-MDT0001 complete [ 2333.012798] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2335.466233] Key type lgssc unregistered [ 2335.763324] LNet: 75050:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2335.769987] LNetError: 75050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2335.787417] LNet: Removed LNI 192.168.201.103@tcp [ 2336.692335] Key type .llcrypt unregistered [ 2336.697562] Key type ._llcrypt unregistered [ 2359.734793] Key type ._llcrypt registered [ 2359.738304] Key type .llcrypt registered [ 2359.832826] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2360.894203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2361.020641] alg: No test for adler32 (adler32-zlib) [ 2362.101183] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 2362.268677] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2363.903165] Key type lgssc registered [ 2365.001817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2372.211077] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2372.222762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2387.746707] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2392.863358] LustreError: 76425:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.103@tcp: failed processing log, type 4: rc = -110 [ 2419.554328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2425.896735] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2431.479203] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2437.988636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2438.085182] Lustre: *** cfs_fail_loc=706, val=0*** [ 2438.465742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2438.514857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2442.127387] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2443.755946] LustreError: 76450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2443.778696] LustreError: 76450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2448.868819] LustreError: 76451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2448.876042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2449.159639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2453.483457] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2458.959863] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2462.692780] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2464.748397] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2464.761856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2464.787668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2469.344325] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2469.349716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2469.370113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2471.098775] Lustre: server umount lustre-OST0000 complete [ 2480.096324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2480.101498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2480.105937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2480.115684] Lustre: Skipped 1 previous similar message [ 2480.125473] Lustre: Skipped 2 previous similar messages [ 2482.908541] Lustre: server umount lustre-MDT0000 complete [ 2485.221903] LustreError: 77403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2485.241253] LustreError: 77403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2486.572644] LustreError: 76432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788603636 with bad export cookie 11634397367047185192 [ 2486.585986] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2486.588928] LustreError: 76432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2487.075669] Lustre: server umount lustre-MDT0001 complete [ 2497.319561] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2500.640660] Key type lgssc unregistered [ 2500.946750] LNet: 79686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2500.966025] LNetError: 79686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2500.991581] LNet: Removed LNI 192.168.201.103@tcp [ 2501.741914] Key type .llcrypt unregistered [ 2501.744330] Key type ._llcrypt unregistered [ 2530.630433] Key type ._llcrypt registered [ 2530.636279] Key type .llcrypt registered [ 2530.840969] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2532.201752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2532.226320] alg: No test for adler32 (adler32-zlib) [ 2533.314179] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 2533.508535] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2535.247135] Key type lgssc registered [ 2536.761252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2544.551913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2544.570446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2550.006651] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2557.029028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2557.568959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2561.722395] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2563.046537] LustreError: 81758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2563.061963] LustreError: 81758:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2567.163784] LustreError: 81761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.171252] LustreError: 81761:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2568.993131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2569.252531] LustreError: 81758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2569.261421] LustreError: 81758:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2569.287881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2574.305167] LustreError: 81758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2574.314544] LustreError: 81758:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2574.857276] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2579.427595] LustreError: 81756:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2579.448662] LustreError: 81756:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2581.964987] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2581.977328] Lustre: MGS: Client aab5999f-841a-45e5-b54e-cbb753136f1a (at 0@lo) reconnecting [ 2582.005376] LustreError: 81772:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2582.190424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2582.284079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2585.986445] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2602.997639] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2603.000496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2603.016342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2604.003826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.005588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2604.037387] Lustre: Skipped 1 previous similar message [ 2606.270334] Lustre: server umount lustre-OST0000 complete [ 2612.194788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2612.202150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2612.207825] Lustre: Skipped 1 previous similar message [ 2612.225513] Lustre: Skipped 1 previous similar message [ 2615.829777] Lustre: server umount lustre-MDT0000 complete [ 2617.313308] LustreError: 81757:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2617.340282] LustreError: 81757:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2619.733266] Lustre: server umount lustre-MDT0001 complete [ 2623.364357] Lustre: server umount MGS complete [ 2634.760964] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 06:23:03 (1788603783) [ 2662.457248] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 2672.029156] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2715.105070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2719.951403] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2729.876573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2730.105324] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2730.150645] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2730.260118] Lustre: lustre-MDT0001: new disk, initializing [ 2730.343754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2730.369638] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2730.373835] Lustre: 88842:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2730.382656] LustreError: 88842:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2734.458118] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2747.795535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2748.055227] Lustre: lustre-OST0000: new disk, initializing [ 2748.061090] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2748.127814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2753.810871] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2754.660703] Lustre: 89912:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000004ec9847f x1875486895546368/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788603920 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2762.012774] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2768.290096] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2768.301544] Lustre: MGS: Client 4cba1be9-340a-4753-9e1c-5f6e17130660 (at 0@lo) reconnecting [ 2768.330936] Lustre: 88129:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2768.354601] LustreError: 88828:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2768.379245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2768.454296] Lustre: lustre-MDT0000: new disk, initializing [ 2768.563671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2768.586862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2768.673719] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2769.645489] Lustre: 90560:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000034145191 x1875486895552896/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788603935 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2773.286902] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2775.007682] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2778.815733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2778.822699] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2778.884033] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2804.704808] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2804.718058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2804.738579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2807.785076] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2809.396532] Lustre: server umount lustre-OST0000 complete [ 2818.034116] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2818.046422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2818.060798] Lustre: Skipped 2 previous similar messages [ 2818.061512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2818.061520] Lustre: Skipped 2 previous similar messages [ 2819.354055] Lustre: server umount lustre-MDT0000 complete [ 2823.140177] LustreError: 88817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2830.940471] Lustre: server umount lustre-MDT0001 complete [ 2836.069143] Lustre: server umount MGS complete [ 2846.920356] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 06:26:36 (1788603996) [ 2862.184864] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2864.730884] Key type lgssc unregistered [ 2864.940907] LNet: 93024:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2864.958243] LNetError: 93024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2864.976699] LNet: Removed LNI 192.168.201.103@tcp [ 2865.918321] Key type .llcrypt unregistered [ 2865.923953] Key type ._llcrypt unregistered [ 2883.692625] Key type ._llcrypt registered [ 2883.697391] Key type .llcrypt registered [ 2883.841418] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2885.275225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2885.381907] alg: No test for adler32 (adler32-zlib) [ 2886.554995] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 2886.759203] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2888.496314] Key type lgssc registered [ 2889.919905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2899.352487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2899.392899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2900.990898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2904.782690] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2911.202834] LustreError: 94232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2911.224517] LustreError: 94232:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2913.552408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2913.936371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2918.573207] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2925.030563] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2928.991838] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2935.891312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2936.290853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2941.940630] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2946.553168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 2946.883510] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2956.768363] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2956.769751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2956.777246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2956.802579] Lustre: Skipped 1 previous similar message [ 2961.889621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2961.897599] Lustre: Skipped 2 previous similar messages [ 2962.411085] Lustre: server umount lustre-OST0000 complete [ 2967.009035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2967.021400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2967.023760] Lustre: Skipped 1 previous similar message [ 2972.130524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2972.144089] Lustre: Skipped 2 previous similar messages [ 2972.868462] Lustre: server umount lustre-MDT0000 complete [ 2976.740780] LustreError: 94214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788604126 with bad export cookie 12614945582807014396 [ 2976.753399] LustreError: 94214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2976.753964] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2977.262984] Lustre: server umount lustre-MDT0001 complete [ 2988.685790] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2991.540263] Key type lgssc unregistered [ 2991.824742] LNet: 97411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2991.829814] LNetError: 97411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2991.846282] LNet: Removed LNI 192.168.201.103@tcp [ 2992.568895] Key type .llcrypt unregistered [ 2992.571636] Key type ._llcrypt unregistered [ 3008.474775] Key type ._llcrypt registered [ 3008.477701] Key type .llcrypt registered [ 3008.562252] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3009.892232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3010.078136] alg: No test for adler32 (adler32-zlib) [ 3011.231562] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 3011.447201] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3013.135293] Key type lgssc registered [ 3014.259427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3020.395625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3020.403066] LustreError: 98476:0:(tgt_mount.c:2501:server_fill_super()) cfs_race id 716 sleeping [ 3020.408461] LustreError: 98478:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 waking [ 3020.413331] LustreError: 98476:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3020.425955] LustreError: 98476:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3020.430111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3020.431538] LustreError: 98476:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3022.082755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3029.246839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3029.603970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3034.489512] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3039.744578] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing lsmod [ 3043.028931] LustreError: 99643:0:(tgt_mount.c:2501:server_fill_super()) cfs_race id 716 sleeping [ 3043.056229] LustreError: 99650:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 waking [ 3043.060643] LustreError: 99643:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 awake: rc=4971 [ 3043.065593] LustreError: 99643:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3043.078101] LustreError: 99643:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3043.088078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3043.547767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3048.930801] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3048.937297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3048.953519] Lustre: Skipped 2 previous similar messages [ 3048.956612] Lustre: Skipped 1 previous similar message [ 3054.051568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3054.061432] Lustre: Skipped 3 previous similar messages [ 3055.081574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3055.091637] Lustre: Skipped 2 previous similar messages [ 3060.194507] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3060.194521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3060.217121] Lustre: Skipped 2 previous similar messages [ 3060.524225] Lustre: server umount lustre-MDT0000 complete [ 3065.132376] LustreError: 98494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788604215 with bad export cookie 14987209682696029670 [ 3065.139146] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3065.148954] LustreError: 98494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3065.314067] LustreError: 99686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3065.315261] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3065.329930] LustreError: 99686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3065.350350] LustreError: 98946:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000005083539e x1875487396746368/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3065.369900] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3065.846618] Lustre: server umount lustre-MDT0001 complete [ 3072.320277] Lustre: server umount lustre-OST0000 complete [ 3082.250039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3082.731546] LustreError: 100717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3082.872887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3087.457558] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3088.359187] LustreError: 100718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3092.461676] LustreError: 100717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3096.130069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3096.575301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3101.245889] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3107.385445] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3111.264857] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3118.886261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3119.439754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3127.163455] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3127.681226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 3133.492764] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3142.113779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3142.134681] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3142.173836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3147.282340] Lustre: server umount lustre-OST0000 complete [ 3152.866310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3152.874537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3152.885526] Lustre: Skipped 1 previous similar message [ 3157.548557] Lustre: server umount lustre-MDT0000 complete [ 3158.501495] LustreError: 100714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3158.518387] LustreError: 100714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3161.146157] LustreError: 100699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788604311 with bad export cookie 14987209682696030370 [ 3161.151930] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3161.163460] LustreError: 100699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3161.465358] Lustre: server umount lustre-MDT0001 complete [ 3173.082479] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3175.942364] Key type lgssc unregistered [ 3176.201364] LNet: 103896:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3176.207552] LNetError: 103896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3177.258930] LNet: Removed LNI 192.168.201.103@tcp [ 3178.063909] Key type .llcrypt unregistered [ 3178.066189] Key type ._llcrypt unregistered [ 3204.542608] Key type ._llcrypt registered [ 3204.544884] Key type .llcrypt registered [ 3204.631437] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3205.563206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3205.626087] alg: No test for adler32 (adler32-zlib) [ 3206.651844] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 3206.858761] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3208.511241] Key type lgssc registered [ 3209.609158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3217.979728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3218.017406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.599725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3224.235719] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3229.156753] LustreError: 105315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3229.173969] LustreError: 105315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3232.812594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3233.147422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3238.226148] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3245.726793] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3250.088114] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3260.555459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3260.820202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3264.977808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 3266.778517] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3272.472555] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3280.954757] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3285.280788] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3290.593121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3290.599498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3290.619156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3293.668710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3293.678487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3293.687740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3296.091280] Lustre: server umount lustre-OST0000 complete [ 3303.904672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3303.912635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3303.923945] Lustre: Skipped 1 previous similar message [ 3303.929044] Lustre: Skipped 1 previous similar message [ 3309.025459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3309.034058] Lustre: Skipped 2 previous similar messages [ 3313.120752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3313.132112] Lustre: Skipped 1 previous similar message [ 3314.143180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3314.444307] Lustre: server umount lustre-MDT0000 complete [ 3318.250208] LustreError: 106521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3318.271343] LustreError: 106521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3318.954217] LustreError: 105295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788604469 with bad export cookie 1777242213617731967 [ 3318.960139] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3318.969301] LustreError: 105295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3319.330992] Lustre: server umount lustre-MDT0001 complete [ 3331.093493] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3333.449234] Key type lgssc unregistered [ 3333.682253] LNet: 108603:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3333.694498] LNetError: 108603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3333.716580] LNet: Removed LNI 192.168.201.103@tcp [ 3334.451579] Key type .llcrypt unregistered [ 3334.455574] Key type ._llcrypt unregistered [ 3351.601890] Key type ._llcrypt registered [ 3351.607213] Key type .llcrypt registered [ 3351.710428] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3353.124418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3353.142087] alg: No test for adler32 (adler32-zlib) [ 3354.380883] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 3354.644339] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3356.319660] Key type lgssc registered [ 3357.819473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3366.424601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3366.457574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3367.961755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3374.120985] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3378.151519] LustreError: 109806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3378.171536] LustreError: 109806:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3383.272311] LustreError: 109807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3383.813746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3384.110882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3388.698306] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3394.311658] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3397.869994] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3405.262362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3405.655887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3411.893465] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3413.883499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3418.525495] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3429.350080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3429.350932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3429.370239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3429.384129] Lustre: Skipped 1 previous similar message [ 3434.466450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3435.159410] Lustre: server umount lustre-OST0000 complete [ 3439.588818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3439.591435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3439.602706] Lustre: Skipped 2 previous similar messages [ 3439.614088] Lustre: Skipped 2 previous similar messages [ 3444.707579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3444.723272] Lustre: Skipped 1 previous similar message [ 3449.825406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3449.842483] Lustre: Skipped 1 previous similar message [ 3453.409500] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3454.945230] LustreError: 110786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3454.964022] LustreError: 110786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3455.701982] Lustre: server umount lustre-MDT0000 complete [ 3459.054739] LustreError: 111961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3459.069427] LustreError: 111961:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3460.030908] LustreError: 109786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788604610 with bad export cookie 8672643883333518416 [ 3460.036069] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3460.050203] LustreError: 109786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3460.445732] Lustre: server umount lustre-MDT0001 complete [ 3471.719523] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3474.778707] Key type lgssc unregistered [ 3475.097880] LNet: 112988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3475.101173] LNetError: 112988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3475.110531] LNet: Removed LNI 192.168.201.103@tcp [ 3475.837916] Key type .llcrypt unregistered [ 3475.843583] Key type ._llcrypt unregistered [ 3504.244931] Key type ._llcrypt registered [ 3504.246636] Key type .llcrypt registered [ 3504.601561] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3506.532440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3506.617310] alg: No test for adler32 (adler32-zlib) [ 3508.410978] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 3508.750462] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3510.567189] Key type lgssc registered [ 3512.235356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3521.871191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3521.888849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3523.542757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3528.716244] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3533.796375] LustreError: 114392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3533.822955] LustreError: 114392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3538.914700] LustreError: 114393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3538.982574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3539.348199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3544.728408] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3550.859466] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3554.698236] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3563.042361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3563.638802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3568.775101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3569.775622] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3575.448802] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3583.099677] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3588.195774] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3588.199070] Lustre: Skipped 2 previous similar messages [ 3589.328180] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3589.383660] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3593.336281] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3593.339603] Lustre: Skipped 2 previous similar messages [ 3600.461546] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3600.471622] Lustre: Skipped 1 previous similar message [ 3606.693282] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.3@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3606.700167] Lustre: Skipped 2 previous similar messages [ 3607.500914] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.3@tcp 192.168.0.[2,10]@tcp [ 3622.109341] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.3/24@tcp in log lustre-MDT0000 [ 3622.118276] Lustre: Skipped 2 previous similar messages [ 3629.139961] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.3/24@tcp [ 3629.146849] Lustre: Skipped 1 previous similar message [ 3636.706433] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3636.724856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3636.748172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3640.802280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3640.812273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3640.825592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3640.857000] Lustre: Skipped 1 previous similar message [ 3642.717897] Lustre: server umount lustre-OST0000 complete [ 3651.045267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3651.047390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3651.052615] Lustre: Skipped 1 previous similar message [ 3653.487394] Lustre: server umount lustre-MDT0000 complete [ 3656.161282] LustreError: 114393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3656.180574] LustreError: 114393:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3656.906308] LustreError: 114374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788604806 with bad export cookie 10751989952186112979 [ 3656.912372] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3656.917238] LustreError: 114374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3657.466767] Lustre: server umount lustre-MDT0001 complete [ 3668.477409] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3671.326624] Key type lgssc unregistered [ 3671.729838] LNet: 118957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3671.739752] LNetError: 118957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3672.807082] LNet: Removed LNI 192.168.201.103@tcp [ 3673.784098] Key type .llcrypt unregistered [ 3673.787276] Key type ._llcrypt unregistered [ 3705.575880] Key type ._llcrypt registered [ 3705.581932] Key type .llcrypt registered [ 3705.718507] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3707.027536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3707.100438] alg: No test for adler32 (adler32-zlib) [ 3708.149573] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 3708.368727] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3710.079137] Key type lgssc registered [ 3711.349511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3721.200274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3721.246209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3722.753045] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3722.784074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3722.818546] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.3/24@tcp [ 3727.277899] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3732.000970] LustreError: 120500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3732.015797] LustreError: 120500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3736.052404] LustreError: 120501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3736.313257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3736.604936] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3736.627826] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.3/24@tcp [ 3736.711159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3741.113139] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3747.616167] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3751.834426] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3758.446677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3758.918374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3765.014734] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3767.117098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 3771.442335] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3781.601888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3781.613704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3781.626262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3782.629417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3782.630837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3782.647618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3787.746413] LustreError: 122103:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3787.768739] LustreError: 122103:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3787.856836] Lustre: server umount lustre-OST0000 complete [ 3792.864187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3792.865983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3792.879159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3792.879166] Lustre: Skipped 1 previous similar message [ 3792.926111] Lustre: Skipped 1 previous similar message [ 3798.009159] LustreError: 120501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3798.039747] LustreError: 120501:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3798.112553] Lustre: server umount lustre-MDT0000 complete [ 3802.631960] LustreError: 120482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788604952 with bad export cookie 5754554298674332042 [ 3802.641903] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3802.646951] LustreError: 120482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3803.076291] Lustre: server umount lustre-MDT0001 complete [ 3815.341959] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3818.503504] Key type lgssc unregistered [ 3818.910707] LNet: 123734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3818.919775] LNetError: 123734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3818.936745] LNet: Removed LNI 192.168.201.103@tcp [ 3819.938861] Key type .llcrypt unregistered [ 3819.942079] Key type ._llcrypt unregistered [ 3849.665416] Key type ._llcrypt registered [ 3849.670896] Key type .llcrypt registered [ 3849.809172] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 3867.133416] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3868.471856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3868.502397] alg: No test for adler32 (adler32-zlib) [ 3869.744873] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 3870.077873] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3871.776456] Key type lgssc registered [ 3872.889814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3911.372238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3911.423506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3912.924951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3912.990796] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3913.076720] Lustre: lustre-MDT0000: new disk, initializing [ 3913.220612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3913.251808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3917.609477] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3928.688259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3928.798338] Lustre: 127105:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3928.844245] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3928.848045] Lustre: Skipped 1 previous similar message [ 3928.908377] Lustre: lustre-MDT0001: new disk, initializing [ 3928.995429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3929.025401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3929.042807] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3933.755579] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3941.992583] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3947.021449] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3953.984462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3954.201265] Lustre: lustre-OST0000: new disk, initializing [ 3954.206477] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3954.287405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3960.350961] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3963.969659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3963.996029] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3964.176717] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3969.115892] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3974.244681] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3974.519179] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3979.613164] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3979.971540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3984.870132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3984.874614] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3984.897991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3989.989490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3990.002561] Lustre: Skipped 2 previous similar messages [ 3995.119170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3995.136922] Lustre: Skipped 1 previous similar message [ 3996.127195] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3996.329966] Lustre: server umount lustre-OST0000 complete [ 4000.744079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4000.768588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4000.801156] Lustre: Skipped 1 previous similar message [ 4000.806449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4005.346771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4005.353457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4005.370220] Lustre: Skipped 1 previous similar message [ 4007.240993] Lustre: server umount lustre-MDT0000 complete [ 4011.466717] LustreError: 127097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788605161 with bad export cookie 17126862078833477256 [ 4011.475956] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4011.479328] LustreError: 127097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4012.209269] Lustre: server umount lustre-MDT0001 complete [ 4021.634605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4022.051570] LustreError: 130920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4022.071588] LustreError: 130920:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4022.130394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4027.362009] LustreError: 130921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4027.530749] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4032.502846] LustreError: 130920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4037.610508] LustreError: 130921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4039.077767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4039.429414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4044.782079] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4050.989451] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4055.298245] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4062.695562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4063.125327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4069.979745] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4076.096551] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4081.322080] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4081.596747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4087.147906] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4087.386633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4096.217269] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4096.448876] Lustre: lustre-OST0001: new disk, initializing [ 4096.460098] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4096.582465] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4103.283904] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4103.830562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4103.844048] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4103.980271] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4113.224906] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4113.503182] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4118.279494] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4118.544537] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4123.217259] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97da42961800.ost_server_uuid 50 [ 4124.853486] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff97da42961800.ost_server_uuid in FULL state after 0 sec [ 4140.003022] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4140.003890] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4140.013592] LustreError: Skipped 1 previous similar message [ 4140.017028] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4140.040318] Lustre: Skipped 1 previous similar message [ 4140.063801] Lustre: Skipped 1 previous similar message [ 4141.461764] Lustre: server umount lustre-OST0001 complete [ 4145.130024] LustreError: 134292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4145.172507] LustreError: 134292:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4146.147473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4146.157338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4152.331940] Lustre: server umount lustre-OST0000 complete [ 4157.408235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4157.423563] LustreError: Skipped 1 previous similar message [ 4157.433747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4157.450127] Lustre: Skipped 1 previous similar message [ 4157.456450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4157.472830] Lustre: Skipped 4 previous similar messages [ 4170.208990] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4170.497965] Lustre: server umount lustre-MDT0000 complete [ 4170.728182] LustreError: 130921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4170.755041] LustreError: 130921:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4175.038805] LustreError: 130900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788605325 with bad export cookie 17126862078833478138 [ 4175.041541] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4175.073424] LustreError: 130900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4175.407547] Lustre: server umount lustre-MDT0001 complete [ 4194.632811] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 4197.816769] Key type lgssc unregistered [ 4198.205452] LNet: 136765:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4198.211808] LNetError: 136765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4198.223234] LNet: Removed LNI 192.168.201.103@tcp [ 4199.207228] Key type .llcrypt unregistered [ 4199.209930] Key type ._llcrypt unregistered [ 4242.423868] Key type ._llcrypt registered [ 4242.425618] Key type .llcrypt registered [ 4242.530345] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 4257.588917] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4259.462584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4259.496565] alg: No test for adler32 (adler32-zlib) [ 4260.565452] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 4260.853958] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 4262.527194] Key type lgssc registered [ 4263.784697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4297.133595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4297.194919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4298.496950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4298.525158] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4298.583756] Lustre: lustre-MDT0000: new disk, initializing [ 4298.650305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4298.670223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4303.132235] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4313.975110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4314.098882] Lustre: 140890:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4314.147574] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4314.151901] Lustre: Skipped 1 previous similar message [ 4314.249734] Lustre: lustre-MDT0001: new disk, initializing [ 4314.329080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4314.375059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4314.392306] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4318.926085] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4326.870437] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4331.920634] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4338.643650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4338.912498] Lustre: lustre-OST0000: new disk, initializing [ 4338.920868] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4339.030802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4345.308445] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4347.998373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4348.011080] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4348.120905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4353.693851] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4358.037482] Lustre: Failing over lustre-OST0000 [ 4358.193351] Lustre: server umount lustre-OST0000 complete [ 4373.985549] Lustre: 139114:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788605508/real 1788605508] req@000000006bd668df x1875488706429312/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788605524 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4374.000863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4374.029143] Lustre: 139114:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4374.073173] Lustre: Skipped 1 previous similar message [ 4377.055613] Lustre: 139115:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788605510/real 1788605510] req@000000002784750a x1875488706429568/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788605526 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4377.109582] Lustre: 139115:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4377.960233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4378.713551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4378.729919] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4379.151187] Lustre: 139116:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788605513/real 1788605513] req@000000009d691083 x1875488706430592/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788605529 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4384.871279] Lustre: 139115:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788605518/real 1788605518] req@00000000e165fd17 x1875488706431488/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788605534 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4384.875874] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4384.898593] Lustre: 139115:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4385.669672] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4387.561194] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4387.563179] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4389.433632] Lustre: Failing over lustre-MDT0000 [ 4389.764202] Lustre: server umount lustre-MDT0000 complete [ 4390.367207] Lustre: 139116:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788605524/real 1788605524] req@00000000bd1ec945 x1875488706432640/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788605540 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4390.381361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4390.390656] Lustre: 139116:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4390.421603] Lustre: Skipped 1 previous similar message [ 4395.489141] LustreError: 141682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4395.512530] LustreError: 141682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4397.765251] LustreError: 140897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4400.608851] LustreError: 140903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4400.623170] LustreError: 140903:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4402.894911] LustreError: 140897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4406.757330] Lustre: 139114:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788605540/real 1788605540] req@00000000dbc7dfa5 x1875488706442240/t0(0) o400->MGC192.168.201.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788605556 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4406.818693] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4408.007470] LustreError: 140898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4408.046284] LustreError: 140898:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4409.502109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4415.969543] LustreError: 139112:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000056c3df4a x1875488706447744/t0(0) o250->MGC192.168.201.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4416.019587] LustreError: 141956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4416.038414] LustreError: 141956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4416.463468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4416.548400] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4418.244416] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4420.749692] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4421.649114] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4421.664173] Lustre: Skipped 1 previous similar message [ 4421.678909] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4421.759150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4431.851982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4431.858934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4431.877310] Lustre: Skipped 2 previous similar messages [ 4431.902385] Lustre: Skipped 1 previous similar message [ 4436.965475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4436.982962] Lustre: Skipped 1 previous similar message [ 4442.087210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4442.097504] Lustre: Skipped 1 previous similar message [ 4444.130991] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4444.466839] Lustre: server umount lustre-OST0000 complete [ 4452.323272] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4452.335336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4452.344845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4455.869835] Lustre: server umount lustre-MDT0000 complete [ 4457.442995] LustreError: 141682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4457.466463] LustreError: 141682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4460.176623] LustreError: 144302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788605610 with bad export cookie 9137581911962972420 [ 4460.183029] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4460.196254] LustreError: 144302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4460.612324] Lustre: server umount lustre-MDT0001 complete [ 4476.238925] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 4479.771753] Key type lgssc unregistered [ 4480.208685] LNet: 146132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4480.217905] LNetError: 146132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4480.243231] LNet: Removed LNI 192.168.201.103@tcp [ 4481.360132] Key type .llcrypt unregistered [ 4481.361896] Key type ._llcrypt unregistered [ 4520.623977] Key type ._llcrypt registered [ 4520.626592] Key type .llcrypt registered [ 4520.741304] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4521.956703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4522.019241] alg: No test for adler32 (adler32-zlib) [ 4523.171657] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 4523.409208] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 4525.151597] Key type lgssc registered [ 4526.208631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4534.338707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4534.359225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4536.002911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4540.283144] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4546.537041] LustreError: 147637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4546.561571] LustreError: 147637:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4548.708771] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4551.652035] LustreError: 147638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4552.924350] LustreError: 147666:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4552.934157] LustreError: 147666:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4552.949465] LustreError: 147666:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4553.252477] Lustre: server umount lustre-MDT0000 complete [ 4571.443219] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 4588.179470] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4626.267602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4626.508978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4626.533291] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4626.588618] Lustre: lustre-MDT0000: new disk, initializing [ 4626.666283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4626.680941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4631.387218] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4642.453210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4642.569129] Lustre: 150847:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4642.661547] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4642.671279] Lustre: Skipped 1 previous similar message [ 4642.722337] Lustre: lustre-MDT0001: new disk, initializing [ 4642.800673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4642.831191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4642.841432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4647.365917] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4655.710492] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4660.527576] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4668.295743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4668.632605] Lustre: lustre-OST0000: new disk, initializing [ 4668.648884] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4668.801571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4675.205482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4675.226955] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4675.396601] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4676.039627] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4685.301053] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4690.652836] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4692.759638] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 06:57:21 (1788605841) [ 4916.201026] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4916.201898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4916.222134] Lustre: Skipped 2 previous similar messages [ 4916.241701] Lustre: Skipped 2 previous similar messages [ 4921.315681] LustreError: 150860:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4923.369404] Lustre: server umount lustre-MDT0000 complete [ 4926.441317] LustreError: 154243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4926.477311] LustreError: 154243:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4927.428428] LustreError: 152599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788606077 with bad export cookie 7517684847737267837 [ 4927.429659] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4927.445270] LustreError: 152599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4931.551856] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4931.556061] LustreError: 150856:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4931.579761] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4934.113338] Lustre: server umount lustre-MDT0001 complete [ 4945.471372] Lustre: server umount lustre-OST0000 complete [ 4956.902647] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 4967.087685] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5004.971240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5005.301199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5005.353406] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5005.460896] Lustre: lustre-MDT0000: new disk, initializing [ 5005.593769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5005.651330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5010.580109] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5021.188540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5021.322548] Lustre: 157111:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5021.394294] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5021.398271] Lustre: Skipped 1 previous similar message [ 5021.494497] Lustre: lustre-MDT0001: new disk, initializing [ 5021.618686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5021.658246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5021.668575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5026.005919] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5033.114970] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5036.738185] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5043.686587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5043.889280] Lustre: lustre-OST0000: new disk, initializing [ 5043.891970] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5043.955343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5048.983668] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5051.447691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5051.463486] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5051.580141] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5055.461418] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5059.660625] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5059.904097] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5064.197894] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5064.447904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5067.235035] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5067.245602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5067.245735] Lustre: Skipped 1 previous similar message [ 5067.245975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5067.261491] Lustre: Skipped 2 previous similar messages [ 5072.362844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5072.373839] Lustre: Skipped 1 previous similar message [ 5072.471681] Lustre: server umount lustre-OST0000 complete [ 5077.478657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5077.488098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5077.508766] Lustre: Skipped 1 previous similar message [ 5077.524555] Lustre: Skipped 1 previous similar message [ 5083.039464] Lustre: server umount lustre-MDT0000 complete [ 5087.452295] LustreError: 157102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788606237 with bad export cookie 7517684847737426667 [ 5087.458268] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5087.466574] LustreError: 157102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5088.025676] Lustre: server umount lustre-MDT0001 complete [ 5097.558543] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:04:06 (1788606246) [ 5113.914072] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 5123.285379] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5160.412740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5160.727091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5160.756972] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5160.864987] Lustre: lustre-MDT0000: new disk, initializing [ 5160.965415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5160.979382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5165.346124] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5176.831230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5177.001824] Lustre: 163430:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5177.305509] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5182.160249] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5190.462807] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5194.288303] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5200.698073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5201.070947] Lustre: lustre-OST0000: new disk, initializing [ 5201.078683] Lustre: Skipped 1 previous similar message [ 5201.090825] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5201.097126] Lustre: Skipped 2 previous similar messages [ 5207.590332] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5210.200845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5210.214196] Lustre: Skipped 1 previous similar message [ 5210.226503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5210.350357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5215.008437] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5225.956583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5225.961649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5225.970774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5225.980904] LustreError: Skipped 1 previous similar message [ 5225.984690] Lustre: Skipped 2 previous similar messages [ 5227.461439] Lustre: server umount lustre-OST0000 complete [ 5233.642552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5236.196859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5236.211878] Lustre: Skipped 2 previous similar messages [ 5237.448768] Lustre: server umount lustre-MDT0000 complete [ 5241.133245] LustreError: 163422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788606391 with bad export cookie 7517684847737427549 [ 5241.135783] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5241.154989] LustreError: 163422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5241.631844] Lustre: server umount lustre-MDT0001 complete [ 5250.103918] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:06:39 (1788606399) [ 5267.497069] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 5280.755972] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5322.156801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5322.386825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5322.417671] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5322.485907] Lustre: lustre-MDT0000: new disk, initializing [ 5322.566182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5322.571814] Lustre: Skipped 2 previous similar messages [ 5322.587341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5327.270701] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5338.698541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5338.848434] Lustre: 169358:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5339.214194] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5345.022368] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5355.452323] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5361.760549] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5369.785063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5375.604645] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5375.650428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5376.266918] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5384.281810] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5396.451621] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5396.460100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5396.462226] LustreError: Skipped 1 previous similar message [ 5396.464855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5396.464862] Lustre: Skipped 4 previous similar messages [ 5396.506868] Lustre: Skipped 1 previous similar message [ 5398.384895] Lustre: server umount lustre-OST0000 complete [ 5405.666497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5408.908331] Lustre: server umount lustre-MDT0000 complete [ 5411.810717] LustreError: 169370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5411.830843] LustreError: 169370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5412.179164] LustreError: 169349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788606562 with bad export cookie 7517684847737428634 [ 5412.186466] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5412.197458] LustreError: 169349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5412.534833] Lustre: server umount lustre-MDT0001 complete [ 5424.637863] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5427.672200] Key type lgssc unregistered [ 5428.057706] LNet: 172834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5428.062799] LNetError: 172834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5428.077772] LNet: Removed LNI 192.168.201.103@tcp [ 5429.038890] Key type .llcrypt unregistered [ 5429.041672] Key type ._llcrypt unregistered [ 5454.758595] Key type ._llcrypt registered [ 5454.764878] Key type .llcrypt registered [ 5454.915027] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5456.215098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5456.275176] alg: No test for adler32 (adler32-zlib) [ 5457.319378] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 5457.592128] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5459.271230] Key type lgssc registered [ 5460.426189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5468.983598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5469.032592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5470.694469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5475.319766] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5480.932253] LustreError: 174233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5480.955026] LustreError: 174233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5485.113880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5485.509083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5490.580982] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5497.618276] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5501.671464] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5509.090320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5509.599456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5516.808112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5516.851688] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5522.577693] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5546.465238] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5546.476357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5546.488338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5547.488533] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5547.512403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5547.516470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5547.532992] Lustre: Skipped 1 previous similar message [ 5549.068220] Lustre: server umount lustre-OST0000 complete [ 5557.216695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5557.228133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5557.253320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5561.247688] Lustre: server umount lustre-MDT0000 complete [ 5562.849622] LustreError: 174233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5562.865235] LustreError: 174233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5566.099214] LustreError: 174214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788606716 with bad export cookie 9731230371269858398 [ 5566.101515] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5566.110278] LustreError: 174214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5566.660922] Lustre: server umount lustre-MDT0001 complete [ 5578.470389] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5582.149802] Key type lgssc unregistered [ 5582.533473] LNet: 177439:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5582.550428] LNetError: 177439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5583.599557] LNet: Removed LNI 192.168.201.103@tcp [ 5584.733291] Key type .llcrypt unregistered [ 5584.738643] Key type ._llcrypt unregistered [ 5613.273870] Key type ._llcrypt registered [ 5613.278481] Key type .llcrypt registered [ 5613.405258] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5615.267719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5615.390349] alg: No test for adler32 (adler32-zlib) [ 5616.608423] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 5616.978795] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5618.807460] Key type lgssc registered [ 5620.529369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5628.466047] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5628.496642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5629.934683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5634.366804] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5639.149257] LustreError: 178845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5639.175575] LustreError: 178845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5642.236597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5642.578687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5646.671094] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5651.763562] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5655.294419] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5663.084910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5663.467822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5670.065236] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5674.757587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5675.100829] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5683.177760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5683.198148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5683.235818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5685.220625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5685.223657] Lustre: Skipped 1 previous similar message [ 5685.234563] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5685.256146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5688.547366] LustreError: 180437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5688.725633] Lustre: server umount lustre-OST0000 complete [ 5695.449156] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff97da49d78800.ost_server_uuid 50 [ 5697.081746] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5710.464564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5710.476960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5710.484208] Lustre: Skipped 2 previous similar messages [ 5716.944200] Lustre: server umount lustre-MDT0000 complete [ 5716.969194] LustreError: 178845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5717.005568] LustreError: 178845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5721.245250] LustreError: 178827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788606871 with bad export cookie 2835249508469009673 [ 5721.265271] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5721.925528] Lustre: server umount lustre-MDT0001 complete [ 5731.028821] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 07:14:39 (1788606879) [ 5737.196335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5737.557032] LustreError: 182327:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5737.650827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5743.051394] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5743.075129] LustreError: 182328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5752.290977] LustreError: 182328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5752.314347] LustreError: 182328:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5752.827820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5753.174409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5756.951718] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5761.678135] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5765.533789] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5771.341393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5771.600454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5776.337783] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5781.347602] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5782.895895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5787.340500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5787.570176] Lustre: lustre-OST0001: new disk, initializing [ 5787.575220] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5787.668455] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5793.278932] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5794.385332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5794.396387] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5794.553865] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5801.314581] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5804.515221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5804.521239] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5804.532668] Lustre: Skipped 1 previous similar message [ 5804.543727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5804.548296] Lustre: Skipped 2 previous similar messages [ 5809.920084] Lustre: server umount lustre-OST0000 complete [ 5810.152173] LustreError: 183924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5810.161658] LustreError: 183924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5816.259519] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5816.507599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5821.354385] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5821.558736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5835.744875] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5835.752871] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5835.768273] Lustre: Skipped 2 previous similar messages [ 5835.792980] Lustre: Skipped 4 previous similar messages [ 5841.560856] Lustre: server umount lustre-OST0001 complete [ 5845.472643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5845.477783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5845.493662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5845.497291] Lustre: Skipped 2 previous similar messages [ 5851.835189] Lustre: server umount lustre-MDT0000 complete [ 5855.817461] LustreError: 183045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788607005 with bad export cookie 2835249508469010744 [ 5855.822794] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5855.844594] LustreError: 183045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5856.181344] Lustre: server umount lustre-MDT0001 complete [ 5880.287886] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 07:17:09 (1788607029) [ 5886.535695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5886.791375] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5886.824802] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5887.009530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5890.884754] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5898.714275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5898.815521] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5898.871484] Lustre: 188122:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5903.926676] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5909.356565] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5912.637964] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5919.405036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5919.564600] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5925.507365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5926.387580] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5933.434756] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5940.629612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5940.808411] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5948.272608] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5950.158211] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5954.167671] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5959.926568] Lustre: lustre-OST0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 5959.942269] Lustre: Skipped 3 previous similar messages [ 5960.679357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5960.697318] Lustre: Skipped 2 previous similar messages [ 5963.245493] Lustre: server umount lustre-OST0000 complete [ 5965.012288] LustreError: 190047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5965.032163] LustreError: 190047:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 5981.157169] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5981.173266] LustreError: Skipped 1 previous similar message [ 5986.182791] Lustre: server umount lustre-OST0001 complete [ 5991.394380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5996.638494] Lustre: server umount lustre-MDT0000 complete [ 6001.307236] LustreError: 188849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788607151 with bad export cookie 2835249508469012151 [ 6001.310368] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6001.324640] LustreError: 188849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6001.812067] Lustre: server umount lustre-MDT0001 complete [ 6025.967836] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 07:19:34 (1788607174) [ 6042.156511] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 6051.590201] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6093.311039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6093.599903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6093.632772] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6093.713329] Lustre: lustre-MDT0000: new disk, initializing [ 6093.830165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6093.839211] Lustre: Skipped 3 previous similar messages [ 6093.851974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6098.228265] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6108.972758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6109.159305] Lustre: 195730:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6109.238690] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6109.243138] Lustre: Skipped 1 previous similar message [ 6109.355602] Lustre: lustre-MDT0001: new disk, initializing [ 6109.518620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6109.545407] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6115.014650] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6121.918216] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6126.169948] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6133.344665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6133.772411] Lustre: lustre-OST0000: new disk, initializing [ 6133.777306] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6142.270425] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6143.030510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6143.048924] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6143.168939] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6150.090397] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6155.353915] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6155.617266] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6160.684237] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6160.855934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6163.936533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6163.937571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6163.961430] LustreError: Skipped 1 previous similar message [ 6163.962901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6163.984214] Lustre: Skipped 5 previous similar messages [ 6163.986575] Lustre: Skipped 6 previous similar messages [ 6177.247167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6177.558710] Lustre: server umount lustre-OST0000 complete [ 6187.962602] Lustre: server umount lustre-MDT0000 complete [ 6191.701496] LustreError: 195723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788607341 with bad export cookie 2835249508469013936 [ 6191.709085] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6191.715567] LustreError: 195723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6192.204843] Lustre: server umount lustre-MDT0001 complete [ 6200.298252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6200.543455] LustreError: 199542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6200.570323] LustreError: 199542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 6205.418236] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6214.519394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6219.742730] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6226.441701] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6231.554207] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6239.837847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6240.299792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6240.308328] Lustre: Skipped 4 previous similar messages [ 6246.467354] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6252.850205] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6258.387228] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6258.604111] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6264.050422] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6264.347365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6269.920199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6269.930846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6269.942034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6269.950566] Lustre: Skipped 3 previous similar messages [ 6269.962334] Lustre: Skipped 11 previous similar messages [ 6272.602064] Lustre: server umount lustre-OST0000 complete [ 6280.642044] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6285.386372] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6314.696227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6321.333792] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6327.685721] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6332.645096] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6335.475748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6335.502417] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6335.510203] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6340.578596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6351.327182] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6351.714936] Lustre: server umount lustre-OST0000 complete [ 6363.682622] Lustre: server umount lustre-MDT0000 complete [ 6366.181781] LustreError: 200937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6366.205374] LustreError: 200937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6368.700050] LustreError: 199524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788607518 with bad export cookie 2835249508469014818 [ 6368.721492] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6369.155652] Lustre: server umount lustre-MDT0001 complete [ 6378.604351] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 07:25:27 (1788607527) [ 6385.906777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6392.359490] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6402.376403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6406.708419] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6412.049811] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6415.984920] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6424.110340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6430.935582] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6437.655809] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6441.838908] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6442.040250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6446.515354] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6446.786731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6455.999903] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6456.298937] Lustre: lustre-OST0001: new disk, initializing [ 6456.308399] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6464.231537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6464.241026] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6464.243373] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6464.428200] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6474.051101] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6479.181293] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6479.578936] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6485.310237] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6485.690664] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6490.082436] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6490.108631] LustreError: Skipped 1 previous similar message [ 6490.111362] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6490.127122] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6490.156425] Lustre: Skipped 5 previous similar messages [ 6490.180303] Lustre: Skipped 12 previous similar messages [ 6502.367761] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6502.719795] Lustre: server umount lustre-OST0001 complete [ 6510.442963] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6510.642363] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6515.767334] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6516.034278] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6545.096238] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6545.306468] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6545.321924] Lustre: Skipped 5 previous similar messages [ 6550.523299] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6550.539406] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6550.548787] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6550.561060] Lustre: Skipped 1 previous similar message [ 6551.640377] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6557.594135] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6559.362784] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6567.611891] Lustre: server umount lustre-OST0001 complete [ 6590.904777] Lustre: server umount lustre-OST0000 complete [ 6601.886624] Lustre: server umount lustre-MDT0000 complete [ 6606.131353] LustreError: 204980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788607756 with bad export cookie 2835249508469015784 [ 6606.136224] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6606.143370] LustreError: 204980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6612.494171] Lustre: server umount lustre-MDT0001 complete [ 6640.074697] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 07:29:48 (1788607788) [ 6648.428663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6648.958413] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6649.054490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6654.040575] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6663.731619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6663.913857] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6663.965266] Lustre: 212956:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6669.071627] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6675.440866] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6679.492065] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6686.363373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6686.585900] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6693.183312] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6699.097467] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6708.717163] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6708.854751] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6716.113419] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6722.794753] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6724.548124] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6729.182520] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6729.482437] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6734.441471] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6734.658649] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6738.965159] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97da44d28800.ost_server_uuid 50 [ 6740.513421] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff97da44d28800.ost_server_uuid in IDLE state after 0 sec [ 6741.636208] Lustre: Permanently deactivating lustre-OST0001 [ 6741.652855] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6745.259653] Lustre: Permanently reactivating lustre-OST0001 [ 6754.277961] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6754.289955] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6754.298343] LustreError: Skipped 3 previous similar messages [ 6754.307163] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6754.318962] Lustre: Skipped 7 previous similar messages [ 6754.331101] Lustre: Skipped 15 previous similar messages [ 6756.023996] Lustre: server umount lustre-OST0001 complete [ 6759.435463] LustreError: 214969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6759.443774] LustreError: 214969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 36 previous similar messages [ 6767.021505] Lustre: server umount lustre-OST0000 complete [ 6778.427613] Lustre: server umount lustre-MDT0000 complete [ 6783.754986] LustreError: 212947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788607933 with bad export cookie 2835249508469017072 [ 6783.759827] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6783.768740] LustreError: 212947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6784.393134] Lustre: server umount lustre-MDT0001 complete [ 6808.702491] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 07:32:37 (1788607957) [ 6816.765502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6817.170871] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6817.233231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6817.241155] Lustre: Skipped 5 previous similar messages [ 6821.644334] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6830.554525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6830.766623] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6830.805256] Lustre: 219062:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6836.472159] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6842.738263] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6847.077632] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6855.724340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6855.901703] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6855.932749] Lustre: Permanently deactivating lustre-OST0000 [ 6855.946878] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6864.972427] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6865.106721] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6873.770381] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6882.223622] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6882.454270] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6890.381696] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6890.747158] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6897.587850] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6900.232582] Lustre: Permanently reactivating lustre-OST0000 [ 6900.254988] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6900.262930] Lustre: Skipped 2 previous similar messages [ 6908.558497] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6908.559141] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 43 seconds. I think it's dead, and I am evicting it. exp 000000005eb20404, cur 1788608058 deadline 1788608045 last 1788608015 [ 6908.566789] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6908.581870] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6908.601708] Lustre: Skipped 1 previous similar message [ 6908.673620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6930.399156] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6930.654745] Lustre: server umount lustre-OST0001 complete [ 6949.343755] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6949.557127] Lustre: server umount lustre-OST0000 complete [ 6959.914892] Lustre: server umount lustre-MDT0000 complete [ 6963.771397] LustreError: 222207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788608113 with bad export cookie 2835249508469018745 [ 6963.784940] LustreError: 222207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6963.785028] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6964.372359] Lustre: server umount lustre-MDT0001 complete [ 6975.509643] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 6978.670631] Key type lgssc unregistered [ 6978.972982] LNet: 223482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6978.996650] LNetError: 223482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6979.030805] LNet: Removed LNI 192.168.201.103@tcp [ 6980.082280] Key type .llcrypt unregistered [ 6980.087414] Key type ._llcrypt unregistered [ 7002.438602] Key type ._llcrypt registered [ 7002.441120] Key type .llcrypt registered [ 7002.543617] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7004.283324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7004.450688] alg: No test for adler32 (adler32-zlib) [ 7005.482267] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 7005.658480] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 7007.311161] Key type lgssc registered [ 7009.325333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7021.316523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7021.354076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7023.024305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7027.959840] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7033.322174] LustreError: 224933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7033.342749] LustreError: 224933:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7036.144492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7036.282817] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7036.317164] Lustre: 224922:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7036.326667] Lustre: Permanently deactivating lustre-MDT0001 [ 7036.337288] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7036.669837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7041.032159] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7046.075692] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7046.660210] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7049.253779] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7056.344480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7056.588380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7061.738851] LustreError: 226547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7061.761823] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 7063.194621] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7067.110947] LustreError: 226547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7068.162723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7069.873441] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7073.251562] LustreError: 226550:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7073.288635] LustreError: 226550:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7077.021387] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7077.327226] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7082.490871] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7082.507757] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7083.770498] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7090.564340] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7094.781657] Lustre: Permanently reactivating lustre-MDT0001 [ 7094.810994] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7094.822149] Lustre: Skipped 2 previous similar messages [ 7096.176839] Lustre: lustre-MDT0001: Client 754d96fa-d404-4fe8-ad44-d113e49c986b (at 192.168.201.3@tcp) reconnecting [ 7100.577704] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7100.604472] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7100.618610] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7100.640421] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7101.140563] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7103.038764] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7109.349576] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7109.607506] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7114.071209] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7114.286510] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7116.987204] Lustre: Permanently deactivating lustre-MDT0001 [ 7117.021958] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7117.029666] Lustre: Skipped 2 previous similar messages [ 7124.626901] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7130.622940] Lustre: server umount lustre-MDT0000 complete [ 7133.672028] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7133.679446] Lustre: Skipped 2 previous similar messages [ 7133.680427] LustreError: 225661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7133.693647] LustreError: 225661:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7134.916490] LustreError: 224915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788608285 with bad export cookie 17368142963537127083 [ 7134.928948] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7135.384646] Lustre: server umount lustre-MDT0001 complete [ 7139.913471] Lustre: server umount lustre-OST0000 complete [ 7153.257441] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 07:38:22 (1788608302) [ 7164.356206] Lustre: server umount lustre-OST0001 complete [ 7170.585186] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 7181.357456] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7220.365818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7220.668938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7220.676615] Lustre: Skipped 2 previous similar messages [ 7220.701290] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7220.758873] Lustre: lustre-MDT0000: new disk, initializing [ 7220.850230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7220.880993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7226.130603] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7238.280409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7238.462968] Lustre: 232582:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7238.557528] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7238.563997] Lustre: Skipped 1 previous similar message [ 7238.670770] Lustre: lustre-MDT0001: new disk, initializing [ 7238.836052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7238.896907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7238.902207] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7243.262170] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7251.474082] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7256.134292] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7265.088885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7265.543580] Lustre: lustre-OST0000: new disk, initializing [ 7265.558708] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7265.704758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7273.156907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7273.164319] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7273.339511] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7274.953867] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7284.890637] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7287.445058] LustreError: 234773:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7297.487086] LustreError: 234773:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7297.535689] LustreError: 232588:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7298.660444] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7298.883797] Lustre: lustre-OST0001: new disk, initializing [ 7298.888173] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7298.974123] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7304.209392] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7307.401025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7307.418659] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7307.541758] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7307.591484] LustreError: 232588:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7311.965488] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7313.454765] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7317.998955] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7318.006027] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7318.015056] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7318.020696] Lustre: Skipped 2 previous similar messages [ 7321.581688] Lustre: server umount lustre-OST0001 complete [ 7323.112947] LustreError: 234311:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7323.127502] LustreError: 234311:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7340.000760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7340.013445] LustreError: Skipped 1 previous similar message [ 7340.021711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7340.047649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7340.050253] Lustre: Skipped 1 previous similar message [ 7343.585527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7343.598684] LustreError: 234310:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7343.604960] Lustre: Skipped 1 previous similar message [ 7343.624735] LustreError: 234310:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 7345.179335] Lustre: server umount lustre-OST0000 complete [ 7351.787628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7351.793165] LustreError: Skipped 1 previous similar message [ 7351.796509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7351.808226] Lustre: Skipped 1 previous similar message [ 7351.821869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7355.555640] Lustre: server umount lustre-MDT0000 complete [ 7359.768092] LustreError: 232573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788608509 with bad export cookie 17368142963537130121 [ 7359.774257] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7359.786103] LustreError: 232573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7374.307671] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7374.767582] Lustre: server umount lustre-MDT0001 complete [ 7388.288475] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7391.051873] Key type lgssc unregistered [ 7391.374373] LNet: 237234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7391.385065] LNetError: 237234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7392.428526] LNet: Removed LNI 192.168.201.103@tcp [ 7393.352599] Key type .llcrypt unregistered [ 7393.356648] Key type ._llcrypt unregistered [ 7437.735529] Key type ._llcrypt registered [ 7437.737765] Key type .llcrypt registered [ 7437.832076] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7439.618725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7439.674678] alg: No test for adler32 (adler32-zlib) [ 7440.790667] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 7441.109618] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 7442.799276] Key type lgssc registered [ 7444.155428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7452.597255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7452.622463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7454.099352] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7454.162922] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7454.386983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7459.737465] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7469.972571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7470.137440] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7470.192469] Lustre: 239430:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7470.506740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7475.537411] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7481.952959] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7486.222736] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7494.098051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7494.453226] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7494.699435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7501.090696] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7501.527065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7506.495214] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7563.260499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7563.274149] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7563.286270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7563.296902] Lustre: Skipped 1 previous similar message [ 7567.153047] Lustre: server umount lustre-OST0000 complete [ 7574.112624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7589.047127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7589.422426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7598.007924] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7603.777668] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7603.807358] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7603.821791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7604.227905] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7644.642800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7644.650133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7644.660219] Lustre: Skipped 1 previous similar message [ 7644.664727] Lustre: Skipped 1 previous similar message [ 7649.961897] Lustre: server umount lustre-OST0000 complete [ 7654.887300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7654.899876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7654.903218] LustreError: Skipped 1 previous similar message [ 7654.908786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7654.929069] Lustre: Skipped 1 previous similar message [ 7654.953691] Lustre: Skipped 2 previous similar messages [ 7660.006172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7660.016729] Lustre: Skipped 1 previous similar message [ 7660.395948] Lustre: server umount lustre-MDT0000 complete [ 7664.702809] LustreError: 239422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788608814 with bad export cookie 4537535966288247139 [ 7664.743081] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7664.751799] LustreError: 239422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7665.206269] Lustre: server umount lustre-MDT0001 complete [ 7679.719456] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7682.663472] Key type lgssc unregistered [ 7683.029851] LNet: 244393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7683.045037] LNetError: 244393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7683.067581] LNet: Removed LNI 192.168.201.103@tcp [ 7684.009542] Key type .llcrypt unregistered [ 7684.011932] Key type ._llcrypt unregistered [ 7709.345490] Key type ._llcrypt registered [ 7709.347847] Key type .llcrypt registered [ 7709.454792] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7710.703646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7711.002564] alg: No test for adler32 (adler32-zlib) [ 7712.468400] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 7712.862416] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 7714.553495] Key type lgssc registered [ 7716.431912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7727.441539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7727.468139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7729.832492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7735.365139] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7739.366085] LustreError: 245790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7739.384521] LustreError: 245790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7744.412080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7744.508191] LustreError: 245791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7744.985782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7750.083742] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7756.779214] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7762.414490] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7771.915604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7772.448781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7774.545738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7778.607468] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7783.981354] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7810.532621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7810.538456] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7810.546610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7810.554676] LustreError: Skipped 1 previous similar message [ 7810.559283] Lustre: Skipped 1 previous similar message [ 7815.657297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7815.666217] Lustre: Skipped 1 previous similar message [ 7816.607548] Lustre: server umount lustre-OST0000 complete [ 7820.768582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7820.788707] Lustre: Skipped 1 previous similar message [ 7820.803718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7820.816987] Lustre: Skipped 1 previous similar message [ 7825.893671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7825.908156] Lustre: Skipped 1 previous similar message [ 7826.576724] Lustre: server umount lustre-MDT0000 complete [ 7830.623339] LustreError: 245770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788608980 with bad export cookie 6644628864329333880 [ 7830.633301] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7831.003771] Lustre: server umount lustre-MDT0001 complete [ 7841.606555] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7844.601899] Key type lgssc unregistered [ 7845.097983] LNet: 249747:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7845.109614] LNetError: 249747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7846.191602] LNet: Removed LNI 192.168.201.103@tcp [ 7847.155388] Key type .llcrypt unregistered [ 7847.157531] Key type ._llcrypt unregistered [ 7864.799303] Key type ._llcrypt registered [ 7864.809308] Key type .llcrypt registered [ 7864.953771] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7866.252502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7866.289233] alg: No test for adler32 (adler32-zlib) [ 7867.551875] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 7867.894237] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 7869.607189] Key type lgssc registered [ 7870.748650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7878.290713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7878.325348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7880.221984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7885.215708] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7890.402180] LustreError: 250951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7890.440278] LustreError: 250951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7894.082481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7894.735726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7899.618211] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7905.508888] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7909.408543] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7916.116646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7916.420187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7921.910352] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7925.642716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7927.361630] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7940.064172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7940.074535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7940.092096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7941.088531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7941.089109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7943.981958] Lustre: server umount lustre-OST0000 complete [ 7951.328302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7951.349112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7951.354250] Lustre: Skipped 2 previous similar messages [ 7954.072805] Lustre: server umount lustre-MDT0000 complete [ 7956.460527] LustreError: 252062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7956.502062] LustreError: 252062:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7958.404574] LustreError: 250930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788609108 with bad export cookie 9612584443721651019 [ 7958.419129] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7958.424598] LustreError: 250930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7958.794989] Lustre: server umount lustre-MDT0001 complete [ 7969.685296] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7973.076497] Key type lgssc unregistered [ 7973.374654] LNet: 254276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7973.394923] LNetError: 254276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7973.416898] LNet: Removed LNI 192.168.201.103@tcp [ 7974.571191] Key type .llcrypt unregistered [ 7974.576665] Key type ._llcrypt unregistered [ 8001.981268] Key type ._llcrypt registered [ 8001.982788] Key type .llcrypt registered [ 8002.086267] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8003.228732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8003.419544] alg: No test for adler32 (adler32-zlib) [ 8004.538324] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 8004.909109] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 8006.567197] Key type lgssc registered [ 8007.843613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8017.836698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8017.861977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8019.792164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8025.211867] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8029.154744] LustreError: 255675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8029.167966] LustreError: 255675:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8034.284481] LustreError: 255676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8034.372451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8034.919423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8039.274559] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8044.573397] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8047.686555] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8054.401834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8054.740535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8060.657208] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8064.025793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8066.198543] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8094.689387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8094.701116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8094.723585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8094.724252] Lustre: Skipped 1 previous similar message [ 8094.733979] Lustre: Skipped 1 previous similar message [ 8099.821374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8099.836214] Lustre: Skipped 1 previous similar message [ 8100.465442] Lustre: server umount lustre-OST0000 complete [ 8104.931532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8104.933883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8104.960546] Lustre: Skipped 1 previous similar message [ 8104.977517] Lustre: Skipped 1 previous similar message [ 8110.052902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8111.240265] Lustre: server umount lustre-MDT0000 complete [ 8115.172739] LustreError: 255676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8115.216758] LustreError: 255676:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8115.764898] LustreError: 255655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788609265 with bad export cookie 1992861973044984518 [ 8115.770912] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8115.778170] LustreError: 255655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8116.375522] Lustre: server umount lustre-MDT0001 complete [ 8131.176735] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 8134.557454] Key type lgssc unregistered [ 8134.706561] LNet: 259629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8134.710264] LNetError: 259629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8134.718595] LNet: Removed LNI 192.168.201.103@tcp [ 8135.559573] Key type .llcrypt unregistered [ 8135.565843] Key type ._llcrypt unregistered [ 8157.310460] Key type ._llcrypt registered [ 8157.313170] Key type .llcrypt registered [ 8157.428282] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8159.343364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8159.414039] alg: No test for adler32 (adler32-zlib) [ 8160.480158] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 8160.794792] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 8162.575935] Key type lgssc registered [ 8164.158096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8172.297682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8172.334028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8173.978702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8178.485434] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8184.306883] LustreError: 260855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8184.339528] LustreError: 260855:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8185.312830] LustreError: 260857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8188.486996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8188.734492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8192.871174] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8197.998418] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8201.167168] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8207.806476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8208.393731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8214.397194] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8220.607744] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8220.735806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8233.954029] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8233.967779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8233.985851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8236.011179] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8236.022606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8236.030654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8236.034896] Lustre: Skipped 1 previous similar message [ 8238.169948] Lustre: server umount lustre-OST0000 complete [ 8245.224644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8245.237868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8245.281612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8247.935556] Lustre: server umount lustre-MDT0000 complete [ 8251.363202] LustreError: 260864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8251.396802] LustreError: 260864:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8251.955499] LustreError: 260819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788609402 with bad export cookie 15974093950232777756 [ 8251.980033] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8252.431594] Lustre: server umount lustre-MDT0001 complete [ 8264.080581] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 8267.233691] Key type lgssc unregistered [ 8267.587619] LNet: 264201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8267.593230] LNetError: 264201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8267.652442] LNet: Removed LNI 192.168.201.103@tcp [ 8268.734345] Key type .llcrypt unregistered [ 8268.738146] Key type ._llcrypt unregistered [ 8282.134139] Key type ._llcrypt registered [ 8282.136404] Key type .llcrypt registered [ 8282.224688] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8300.239842] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 8316.056688] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8317.993338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8318.109211] alg: No test for adler32 (adler32-zlib) [ 8319.331226] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 8319.607856] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 8321.376105] Key type lgssc registered [ 8322.944719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8370.292608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8370.332712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8371.653089] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8371.695676] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8371.791953] Lustre: lustre-MDT0000: new disk, initializing [ 8371.894870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8371.914996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8377.670192] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8388.609597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8388.769200] Lustre: 267703:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8388.831876] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8388.842782] Lustre: Skipped 1 previous similar message [ 8388.932216] Lustre: lustre-MDT0001: new disk, initializing [ 8389.072045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8389.119763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8389.131715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8394.129063] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8403.949868] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8409.469299] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8419.172414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8419.509915] Lustre: lustre-OST0000: new disk, initializing [ 8419.519673] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8419.627245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8427.362483] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8428.149146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8428.155123] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8428.275652] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8435.219657] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8440.544541] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8440.816684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8446.633684] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8446.937960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8454.112749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8454.117419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8454.134558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8454.145147] Lustre: Skipped 1 previous similar message [ 8454.160368] Lustre: Skipped 1 previous similar message [ 8455.850970] Lustre: server umount lustre-OST0000 complete [ 8464.358884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8464.377894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8464.387567] Lustre: Skipped 1 previous similar message [ 8467.961698] Lustre: server umount lustre-MDT0000 complete [ 8472.353442] LustreError: 267695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788609622 with bad export cookie 9406532072289853363 [ 8472.366096] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8472.374537] LustreError: 267695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8472.996518] Lustre: server umount lustre-MDT0001 complete [ 8482.559811] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 08:00:31 (1788609631) [ 8490.367858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8491.099938] LustreError: 271713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8491.115090] LustreError: 271713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8491.174133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8495.675721] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8496.608285] LustreError: 271714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8501.742222] LustreError: 271713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8504.411324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8504.871987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8509.197342] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8515.189814] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8519.120521] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8526.689570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8527.105855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8534.531182] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8541.212795] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8554.979099] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8554.987938] LustreError: Skipped 1 previous similar message [ 8555.002690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8555.030986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8555.037894] Lustre: Skipped 1 previous similar message [ 8557.055950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8557.064730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8557.070367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8557.094433] Lustre: Skipped 1 previous similar message [ 8558.940848] Lustre: server umount lustre-OST0000 complete [ 8566.241747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8566.260296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8566.298391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8569.372724] Lustre: server umount lustre-MDT0000 complete [ 8572.402253] LustreError: 271709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8572.426801] LustreError: 271709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8574.375849] LustreError: 271695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788609724 with bad export cookie 9406532072289854245 [ 8574.404709] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8574.751385] Lustre: server umount lustre-MDT0001 complete [ 8587.449236] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 8590.477829] Key type lgssc unregistered [ 8590.781260] LNet: 274911:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8590.788530] LNetError: 274911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8590.800925] LNet: Removed LNI 192.168.201.103@tcp [ 8591.592119] Key type .llcrypt unregistered [ 8591.596676] Key type ._llcrypt unregistered [ 8620.805745] Key type ._llcrypt registered [ 8620.808433] Key type .llcrypt registered [ 8620.946622] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 8636.564944] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8637.539610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8637.566895] alg: No test for adler32 (adler32-zlib) [ 8638.695334] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 8638.897914] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 8640.551165] Key type lgssc registered [ 8641.874507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8691.555600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8691.588334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8692.892844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8692.926237] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8693.017718] Lustre: lustre-MDT0000: new disk, initializing [ 8693.145052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8693.168963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8699.192846] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8712.044632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8712.244482] Lustre: 278703:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8712.326804] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8712.340762] Lustre: Skipped 1 previous similar message [ 8712.552404] Lustre: lustre-MDT0001: new disk, initializing [ 8712.669564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8712.714483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8712.731387] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8717.765878] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8728.347462] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8733.582880] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8742.474350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8742.772847] Lustre: lustre-OST2710: new disk, initializing [ 8742.778468] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8742.886384] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8749.764608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8749.797192] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8749.975216] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8752.644697] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8762.291266] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8769.770600] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8769.992896] Lustre: lustre-OST03e8: new disk, initializing [ 8769.997310] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8770.102724] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8776.246440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8776.265036] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8776.406129] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8777.313202] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8784.411428] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8794.315156] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8794.533743] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8799.758641] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8800.154748] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8805.150655] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8805.299862] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8809.258936] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8809.491623] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8815.073458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8815.086948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8815.115053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8815.765810] Lustre: server umount lustre-MDT0000 complete [ 8817.654160] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8817.686971] Lustre: Skipped 2 previous similar messages [ 8820.035109] LustreError: 278696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788609970 with bad export cookie 10849588100362630215 [ 8820.059792] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8820.069417] LustreError: 278696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8820.467511] Lustre: server umount lustre-MDT0001 complete [ 8834.923483] Lustre: server umount lustre-OST2710 complete [ 8838.096390] Lustre: 276500:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788609972/real 1788609972] req@0000000071dd16dd x1875493297133056/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788609988 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8838.154576] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8840.020701] Lustre: server umount lustre-OST03e8 complete [ 8862.855923] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 8876.344516] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8922.410225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8922.741887] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8922.813797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8922.905258] Lustre: lustre-MDT0000: new disk, initializing [ 8923.073779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8923.129499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8928.181492] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8940.853206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8941.039470] Lustre: 286436:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8941.137929] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8941.141801] Lustre: Skipped 1 previous similar message [ 8941.252153] Lustre: lustre-MDT0001: new disk, initializing [ 8941.468850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8941.568697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8941.605303] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8947.164278] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8955.493289] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8960.598530] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8969.083251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8969.354189] Lustre: lustre-OST0000: new disk, initializing [ 8969.367864] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8969.534722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8976.000093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8976.021946] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8976.233502] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8976.488786] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8985.047754] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8990.854984] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8991.097592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8996.342200] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8996.657949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9001.958275] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9001.961594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9001.979762] LustreError: Skipped 1 previous similar message [ 9001.980478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9002.051397] Lustre: Skipped 1 previous similar message [ 9007.078417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9007.085876] Lustre: Skipped 2 previous similar messages [ 9012.206399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9012.214664] Lustre: Skipped 1 previous similar message [ 9013.221547] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9013.626545] Lustre: server umount lustre-OST0000 complete [ 9018.337258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9018.350296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9018.372174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9027.554834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9027.562663] Lustre: Skipped 3 previous similar messages [ 9032.161445] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9032.417578] Lustre: server umount lustre-MDT0000 complete [ 9032.675635] LustreError: 286448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9032.692214] LustreError: 286448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 9036.541097] LustreError: 286429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788610186 with bad export cookie 10849588100362631895 [ 9036.545699] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9036.555606] LustreError: 286429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9037.061767] Lustre: server umount lustre-MDT0001 complete [ 9047.200599] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 08:09:55 (1788610195) [ 9049.700127] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9051.974570] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 08:10:00 (1788610200) [ 9056.563577] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 9088.915559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9089.291749] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9089.323303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9089.527916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9094.479916] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9104.417485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9104.478161] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9104.502216] Lustre: 291873:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9110.730350] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9118.504312] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9122.966364] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9143.744438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9144.036836] Lustre: Denying initial registration attempt from nid 192.168.201.103@tcp, specified as failover [ 9144.057854] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9144.071844] LustreError: 293490:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -99 [ 9144.081870] LustreError: 293490:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 9144.089544] LustreError: 293490:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 9144.231349] Lustre: server umount lustre-OST0000 complete [ 9144.236717] LustreError: 293490:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9150.947412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9150.947596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9150.949455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9150.957163] Lustre: Skipped 2 previous similar messages [ 9165.279314] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9165.739434] Lustre: server umount lustre-MDT0000 complete [ 9166.307058] LustreError: 291881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9166.327542] LustreError: 291881:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9169.819796] LustreError: 291864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788610319 with bad export cookie 10849588100362632777 [ 9169.832140] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9169.834348] LustreError: 291864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9170.258919] Lustre: server umount lustre-MDT0001 complete [ 9182.558529] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 9185.690865] Key type lgssc unregistered [ 9186.076805] LNet: 294595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9186.084605] LNetError: 294595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9186.118686] LNet: Removed LNI 192.168.201.103@tcp [ 9187.071637] Key type .llcrypt unregistered [ 9187.075480] Key type ._llcrypt unregistered [ 9201.207549] Key type ._llcrypt registered [ 9201.209223] Key type .llcrypt registered [ 9201.323878] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 9202.741897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9202.769044] alg: No test for adler32 (adler32-zlib) [ 9204.021293] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 9204.364216] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 9206.079317] Key type lgssc registered [ 9208.102650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9266.814968] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 9275.901433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9275.961333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9277.685979] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9277.725679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9278.008444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9284.404400] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9294.468670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9294.694500] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9294.800285] Lustre: 297336:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9295.107542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9300.496226] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9308.694914] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9312.811498] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9332.777073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9333.036854] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9333.227685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9339.707752] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9345.351611] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9348.585694] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9348.593355] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9348.614697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9349.090088] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9353.877736] Lustre: server umount lustre-OST0000 complete [ 9359.328929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9359.335374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9359.343598] Lustre: Skipped 2 previous similar messages [ 9359.358906] Lustre: Skipped 3 previous similar messages [ 9364.450919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9364.466415] Lustre: Skipped 1 previous similar message [ 9365.049140] Lustre: server umount lustre-MDT0000 complete [ 9369.730799] LustreError: 297326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788610519 with bad export cookie 5054847444891488933 [ 9369.738660] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9369.745339] LustreError: 297326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9370.222858] Lustre: server umount lustre-MDT0001 complete [ 9383.384064] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 9386.036152] Key type lgssc unregistered [ 9386.313210] LNet: 300552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9386.321950] LNetError: 300552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9386.339412] LNet: Removed LNI 192.168.201.103@tcp [ 9387.213607] Key type .llcrypt unregistered [ 9387.215688] Key type ._llcrypt unregistered [ 9413.477834] Key type ._llcrypt registered [ 9413.479367] Key type .llcrypt registered [ 9413.601889] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 9414.728527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9414.741870] alg: No test for adler32 (adler32-zlib) [ 9415.891741] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 9416.099174] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 9417.824148] Key type lgssc registered [ 9419.271264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9428.080390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9428.099446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9429.721207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9434.165682] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9439.207990] LustreError: 301954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9439.219086] LustreError: 301954:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9442.652814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9442.985241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9447.524301] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9453.463634] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9457.290781] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9466.857987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9467.234414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9474.737335] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9480.628125] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9489.378688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9489.389937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9489.410793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9489.892046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9489.903589] Lustre: Skipped 1 previous similar message [ 9493.712502] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 9493.718424] Lustre: Skipped 2 previous similar messages [ 9495.010324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9495.493422] Lustre: server umount lustre-MDT0000 complete [ 9498.827401] LustreError: 303609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9499.983452] LustreError: 301933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788610650 with bad export cookie 5450452021377783416 [ 9499.987842] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9500.007551] LustreError: 301933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9500.130636] LustreError: 301949:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9500.136186] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9500.175784] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9500.190513] Lustre: Skipped 2 previous similar messages [ 9503.945422] LustreError: 303609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9506.441990] Lustre: server umount lustre-MDT0001 complete [ 9513.579577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9524.220370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9525.407123] LustreError: 305197:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 9525.421325] LustreError: 305197:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@000000008cd89429 x1875494112893184/t0(0) o250->MGC192.168.201.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788610675 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9525.442863] LustreError: 305197:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 9526.061915] LustreError: 303558:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9526.086448] LustreError: 303558:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9526.163141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9526.224458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9531.697341] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9534.379307] LustreError: 305209:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9534.382656] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9534.405077] LustreError: 305209:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9541.310534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9541.838406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9547.121880] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9547.254582] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9552.859877] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9554.482388] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9558.298329] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9560.077855] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9567.200358] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9567.202451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9567.216577] LustreError: Skipped 1 previous similar message [ 9567.229190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9567.238840] Lustre: Skipped 1 previous similar message [ 9567.269433] Lustre: Skipped 3 previous similar messages [ 9571.901981] Lustre: server umount lustre-OST0000 complete [ 9577.449970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9577.457645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9577.477950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9582.741516] Lustre: server umount lustre-MDT0000 complete [ 9583.075119] LustreError: 305211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9583.109506] LustreError: 305211:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9586.772373] LustreError: 303570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788610736 with bad export cookie 5450452021377791018 [ 9586.778923] LustreError: 303570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9586.779437] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9587.299739] Lustre: server umount lustre-MDT0001 complete [ 9598.625912] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 9601.401297] Key type lgssc unregistered [ 9601.631087] LNet: 307644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9601.636091] LNetError: 307644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9601.664123] LNet: Removed LNI 192.168.201.103@tcp [ 9602.654818] Key type .llcrypt unregistered [ 9602.657141] Key type ._llcrypt unregistered [ 9632.362460] Key type ._llcrypt registered [ 9632.364066] Key type .llcrypt registered [ 9632.516046] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 9634.043086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9634.263141] alg: No test for adler32 (adler32-zlib) [ 9635.850608] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 9636.356647] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 9638.161053] Key type lgssc registered [ 9639.616967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9649.145244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9649.167281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9650.671608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9655.812164] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9660.899963] LustreError: 309045:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9660.916178] LustreError: 309045:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9665.663499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9665.966584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9670.228668] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9676.637432] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9681.192251] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9685.472437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9685.490278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9685.497262] Lustre: Skipped 1 previous similar message [ 9690.599044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9690.613669] Lustre: Skipped 1 previous similar message [ 9695.712688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9695.728807] Lustre: Skipped 1 previous similar message [ 9699.299125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9700.833362] LustreError: 309041:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9700.853449] LustreError: 309041:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9701.597610] Lustre: server umount lustre-MDT0000 complete [ 9705.954589] LustreError: 309757:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9705.975217] LustreError: 309757:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9706.379101] LustreError: 309025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788610856 with bad export cookie 107273423028440786 [ 9706.412331] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9706.824951] Lustre: server umount lustre-MDT0001 complete [ 9724.243483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9724.388313] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9724.404907] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9724.521706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9730.595209] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9739.252542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9739.421889] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9739.472637] Lustre: 311407:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9739.867624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9744.020677] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9749.588479] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9753.389967] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9764.013431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9764.280269] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9764.290067] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9764.301983] LustreError: 313066:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -2 [ 9764.316343] LustreError: 313066:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 9764.321795] LustreError: 313066:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 9764.388388] Lustre: server umount lustre-OST0000 complete [ 9764.390684] LustreError: 313066:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9776.053651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9776.318888] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9776.582954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9783.713784] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9785.008101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9790.155323] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9799.191410] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9799.349698] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9799.445416] Lustre: lustre-OST0001: new disk, initializing [ 9799.452915] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9799.567781] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9806.013418] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9807.013206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9807.024268] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9807.296632] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9812.445690] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9817.573947] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9817.574881] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9817.590604] Lustre: Skipped 1 previous similar message [ 9817.597451] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9817.615869] Lustre: Skipped 1 previous similar message [ 9821.575668] Lustre: server umount lustre-OST0001 complete [ 9822.705608] LustreError: 314876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9822.725049] LustreError: 314876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9826.273520] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9826.282363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9826.287057] LustreError: Skipped 1 previous similar message [ 9826.287711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9826.328819] Lustre: Skipped 2 previous similar messages [ 9827.810949] LustreError: 314876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9827.852089] LustreError: 314876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9838.058934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9838.059452] LustreError: 313533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9838.070195] Lustre: Skipped 6 previous similar messages [ 9838.084353] LustreError: 313533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9840.095146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9840.425640] Lustre: server umount lustre-OST0000 complete [ 9847.267941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9847.271677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9850.739206] Lustre: server umount lustre-MDT0000 complete [ 9854.860219] LustreError: 311400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788611004 with bad export cookie 107273423028441339 [ 9854.864514] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9854.884805] LustreError: 311400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9855.368471] Lustre: server umount lustre-MDT0001 complete [ 9866.062311] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 9868.870640] Key type lgssc unregistered [ 9869.262798] LNet: 316324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9869.279214] LNetError: 316324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9869.296883] LNet: Removed LNI 192.168.201.103@tcp [ 9870.256900] Key type .llcrypt unregistered [ 9870.260419] Key type ._llcrypt unregistered [ 9942.930311] Key type ._llcrypt registered [ 9942.936202] Key type .llcrypt registered [ 9943.090191] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 9961.094320] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 9962.713690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9962.755401] alg: No test for adler32 (adler32-zlib) [ 9963.879601] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 9964.115552] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 9965.824411] Key type lgssc registered [ 9967.235396] Lustre: Echo OBD driver; http://www.lustre.org/ [10008.561722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10008.578477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10009.800827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10009.846433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10009.905610] Lustre: lustre-MDT0000: new disk, initializing [10009.986829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10009.999765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10015.029893] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10028.272299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10028.474725] Lustre: 321228:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10028.583760] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10028.590879] Lustre: Skipped 1 previous similar message [10028.806368] Lustre: lustre-MDT0001: new disk, initializing [10029.011338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10029.113610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10029.132318] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10034.203566] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10041.477475] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10045.779444] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10052.991978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10053.306099] Lustre: lustre-OST0000: new disk, initializing [10053.312938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10053.423472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10059.530804] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10059.917808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10059.930650] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10060.045409] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10068.201540] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10073.156729] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10073.453859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10079.778165] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10080.093635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10085.864875] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10085.867207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10085.889595] Lustre: Skipped 1 previous similar message [10090.988019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10091.025283] Lustre: Skipped 2 previous similar messages [10096.111863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10096.125645] Lustre: Skipped 1 previous similar message [10097.121627] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10097.556570] Lustre: server umount lustre-OST0000 complete [10105.824832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10105.830809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10105.860033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10108.220770] Lustre: server umount lustre-MDT0000 complete [10112.029719] LustreError: 322005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788611262 with bad export cookie 402220463767069229 [10112.036185] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10112.044816] LustreError: 322005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10112.593571] Lustre: server umount lustre-MDT0001 complete [10121.261402] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 08:27:50 (1788611270) [10128.238530] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 08:27:57 (1788611277) [10137.407818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10137.964682] LustreError: 325581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10137.989920] LustreError: 325581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10138.075649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10142.778773] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10143.203127] LustreError: 325582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10143.214358] LustreError: 325582:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10148.321344] LustreError: 325581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10151.251975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10151.620827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10156.735186] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10163.244571] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10167.850229] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10174.183744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10174.527466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10181.555887] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10187.705378] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10195.752529] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10198.320682] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10200.168696] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10202.851588] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10205.291501] Lustre: Failing over lustre-MDT0000 [10206.087079] Lustre: server umount lustre-MDT0000 complete [10208.231092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10208.248719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10208.268197] Lustre: Skipped 1 previous similar message [10208.274598] LustreError: 325582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10214.881287] LustreError: 325578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10214.899510] LustreError: 325578:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [10225.119441] Lustre: 319450:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788611359/real 1788611359] req@00000000249cff73 x1875494687548672/t0(0) o400->MGC192.168.201.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788611375 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10225.120842] LustreError: 326292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10225.148482] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10225.184555] LustreError: 326292:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [10225.952983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10235.376888] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x594f961783aa9b9 [10235.390310] Lustre: MGC192.168.201.103@tcp: Connection restored to 0@lo (at 0@lo) [10235.781992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10235.863674] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10236.178826] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10240.145701] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10241.017862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10241.053956] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10241.109186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10248.427125] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10249.916600] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10253.740061] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10256.060909] Lustre: Failing over lustre-MDT0000 [10256.352919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10256.360123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10256.367828] LustreError: 327544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10256.379822] Lustre: Skipped 4 previous similar messages [10256.400580] LustreError: 327544:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [10256.447454] Lustre: server umount lustre-MDT0000 complete [10276.175216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10276.271721] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10276.478722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10276.524937] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10278.597946] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10280.993904] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10281.972729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10281.987430] Lustre: Skipped 2 previous similar messages [10282.037453] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10282.098761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10288.805480] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10290.526310] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10302.435764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10302.436884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10302.442062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10302.462237] Lustre: Skipped 2 previous similar messages [10302.475544] Lustre: Skipped 3 previous similar messages [10304.355497] Lustre: server umount lustre-MDT0000 complete [10307.564889] LustreError: 325577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10307.581366] LustreError: 325577:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [10308.599947] LustreError: 325563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788611458 with bad export cookie 402220463767071945 [10308.601941] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10308.621996] LustreError: 325563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10309.208403] Lustre: server umount lustre-MDT0001 complete [10324.536184] Lustre: server umount lustre-OST0000 complete [10341.416272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10342.161517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10348.903822] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10359.221670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10364.744385] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10372.067199] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10376.279351] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10384.439479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10387.991104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10391.800510] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10398.027951] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10414.562126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10414.562210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10414.564538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10414.564546] Lustre: Skipped 1 previous similar message [10414.580660] Lustre: Skipped 1 previous similar message [10420.038706] Lustre: server umount lustre-OST0000 complete [10430.860764] Lustre: server umount lustre-MDT0000 complete [10435.042948] LustreError: 332386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10435.070734] LustreError: 332386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10435.451146] LustreError: 333833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788611585 with bad export cookie 402220463767072694 [10435.457828] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10435.459550] LustreError: 333833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10436.220535] Lustre: server umount lustre-MDT0001 complete [10448.945899] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [10452.685489] Key type lgssc unregistered [10453.109633] LNet: 334567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10453.128683] LNetError: 334567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10453.155276] LNet: Removed LNI 192.168.201.103@tcp [10454.396301] Key type .llcrypt unregistered [10454.398507] Key type ._llcrypt unregistered [10488.883601] Key type ._llcrypt registered [10488.886743] Key type .llcrypt registered [10489.071261] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [10490.836854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10490.910678] alg: No test for adler32 (adler32-zlib) [10492.588064] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [10492.928085] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [10494.735514] Key type lgssc registered [10496.280915] Lustre: Echo OBD driver; http://www.lustre.org/ [10506.652863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10506.697376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10508.759322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10514.242779] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10517.988906] LustreError: 336065:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10517.999326] LustreError: 336065:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10523.105151] LustreError: 336066:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10524.075681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10524.642849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10529.459992] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10537.192948] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10541.326821] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10548.302632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10548.868340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10552.963786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10556.294839] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10562.243682] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10575.841104] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10575.858195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10575.872169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10578.913602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10578.919457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10578.958259] Lustre: Skipped 1 previous similar message [10581.053857] Lustre: server umount lustre-MDT0000 complete [10584.037882] LustreError: 336777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10584.061501] LustreError: 336777:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10585.103231] LustreError: 336046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788611735 with bad export cookie 11265785901059184661 [10585.109596] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10585.116644] LustreError: 336046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10585.407930] Lustre: server umount lustre-MDT0001 complete [10600.180456] Lustre: server umount lustre-OST0000 complete [10614.165361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10614.866687] LustreError: 339457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10614.988634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10620.384418] LustreError: 339458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10620.963906] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10629.610770] LustreError: 339457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10629.639386] LustreError: 339457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10630.652328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10631.055412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10636.238600] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10643.193225] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10648.199621] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10656.824782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10657.462441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10659.526839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10664.263740] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10670.141159] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10687.458183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10687.472862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10687.482026] Lustre: Skipped 1 previous similar message [10687.497026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10687.497885] Lustre: Skipped 3 previous similar messages [10690.016619] Lustre: server umount lustre-MDT0000 complete [10692.581039] LustreError: 339452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10692.607208] LustreError: 339452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10694.426154] LustreError: 339437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788611844 with bad export cookie 11265785901059185781 [10694.443575] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10694.876773] Lustre: server umount lustre-MDT0001 complete [10710.163823] Lustre: server umount lustre-OST0000 complete [10727.545451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10728.126744] LustreError: 343034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10728.244473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10733.260369] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10743.197387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10748.553182] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10755.885704] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10760.842184] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10770.801408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10771.390783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10771.397627] Lustre: Skipped 1 previous similar message [10774.449299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10780.512975] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10789.452238] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10799.072052] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10799.074045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10799.108904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10799.108911] Lustre: Skipped 1 previous similar message [10799.169959] Lustre: Skipped 2 previous similar messages [10802.831308] Lustre: server umount lustre-OST0000 complete [10809.314202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10809.320224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10809.326150] Lustre: Skipped 1 previous similar message [10809.343028] Lustre: Skipped 2 previous similar messages [10813.230614] Lustre: server umount lustre-MDT0000 complete [10814.443955] LustreError: 343034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10814.479439] LustreError: 343034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10817.739264] LustreError: 343016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788611967 with bad export cookie 11265785901059186789 [10817.739994] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10817.755748] LustreError: 343016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10818.335765] Lustre: server umount lustre-MDT0001 complete [10833.266299] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [10837.777654] Key type lgssc unregistered [10838.381227] LNet: 346217:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10838.399712] LNetError: 346217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10838.450473] LNet: Removed LNI 192.168.201.103@tcp [10839.356949] Key type .llcrypt unregistered [10839.359216] Key type ._llcrypt unregistered [10870.393282] Key type ._llcrypt registered [10870.394777] Key type .llcrypt registered [10870.606649] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [10872.536355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10872.575414] alg: No test for adler32 (adler32-zlib) [10873.677995] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [10873.907783] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [10875.623189] Key type lgssc registered [10877.056105] Lustre: Echo OBD driver; http://www.lustre.org/ [10886.998570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10887.024178] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10887.035976] LustreError: 347646:0:(osd_handler.c:8532:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10887.069747] LustreError: 347646:0:(obd_config.c:843:class_setup()) setup lustre-MDT0000-osd failed (-22) [10887.074947] LustreError: 347646:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10887.081184] LustreError: 347646:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10887.091524] LustreError: 347646:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10895.254750] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10895.265161] LustreError: 348043:0:(osd_handler.c:8532:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10895.292824] LustreError: 348043:0:(obd_config.c:843:class_setup()) setup lustre-OST0000-osd failed (-22) [10895.299357] LustreError: 348043:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10895.304319] LustreError: 348043:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10895.310420] LustreError: 348043:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10917.954717] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [10921.303804] Key type lgssc unregistered [10921.644778] LNet: 349030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10921.649583] LNetError: 349030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10921.689357] LNet: Removed LNI 192.168.201.103@tcp [10922.680898] Key type .llcrypt unregistered [10922.687348] Key type ._llcrypt unregistered [10941.898325] Key type ._llcrypt registered [10941.902315] Key type .llcrypt registered [10942.036721] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [10958.933381] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [10960.143814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10960.151076] alg: No test for adler32 (adler32-zlib) [10961.553609] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [10961.866281] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [10963.599184] Key type lgssc registered [10964.973737] Lustre: Echo OBD driver; http://www.lustre.org/ [11006.683864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11006.725940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11007.994423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11008.023464] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11008.095440] Lustre: lustre-MDT0000: new disk, initializing [11008.161082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11008.183844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11012.715864] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11024.258988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11024.452280] Lustre: 352157:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11024.526257] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11024.530406] Lustre: Skipped 1 previous similar message [11024.597943] Lustre: lustre-MDT0001: new disk, initializing [11024.699154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11024.731030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11024.734898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11028.757988] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11036.800817] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11041.430904] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11049.131744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11049.349823] Lustre: lustre-OST0000: new disk, initializing [11049.355794] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11049.426054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11056.001176] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11057.790347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11057.810200] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11058.032581] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11064.040927] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11068.702874] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11068.929759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11073.175691] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11073.439652] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11078.637209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11078.646221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11078.648195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11078.675223] Lustre: Skipped 1 previous similar message [11081.597837] Lustre: server umount lustre-OST0000 complete [11086.305916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11086.322893] LustreError: Skipped 1 previous similar message [11086.324899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11086.336068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11086.339056] Lustre: Skipped 1 previous similar message [11088.865295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11088.867618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11091.816397] Lustre: server umount lustre-MDT0000 complete [11095.865215] LustreError: 352147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788612245 with bad export cookie 3196147887592640604 [11095.874105] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11095.883286] LustreError: 352147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11096.901717] Lustre: server umount lustre-MDT0001 complete [11106.856930] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11108.469900] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 08:44:17 (1788612257) [11115.099790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11115.695567] LustreError: 356211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11115.718344] LustreError: 356211:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11115.786393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11120.835659] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11121.120718] LustreError: 356212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11125.235483] LustreError: 356211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11129.999697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11130.314892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11134.924563] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11140.596312] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11145.498572] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11154.957468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11155.331369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11161.768424] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11168.429761] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11177.359572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11177.551300] Lustre: lustre-OST0001: new disk, initializing [11177.555796] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11177.669558] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11184.785209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11184.796217] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11184.911133] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11185.822305] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11195.316443] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11200.484546] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11200.493668] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11200.519768] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11200.527287] Lustre: Skipped 1 previous similar message [11200.540913] Lustre: Skipped 2 previous similar messages [11205.613179] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11205.628406] Lustre: Skipped 2 previous similar messages [11206.487404] Lustre: server umount lustre-OST0001 complete [11207.379859] LustreError: 357804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11215.842885] LustreError: 357806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11215.863890] LustreError: 357806:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11226.090681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11226.092606] LustreError: 359005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11226.120815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11226.144484] LustreError: 359005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11226.167675] Lustre: Skipped 1 previous similar message [11231.988512] Lustre: server umount lustre-OST0000 complete [11237.858788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11237.865984] LustreError: Skipped 1 previous similar message [11237.881206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11237.898191] Lustre: Skipped 1 previous similar message [11243.514341] Lustre: server umount lustre-MDT0000 complete [11246.560228] LustreError: 356212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11246.589203] LustreError: 356212:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11247.621243] LustreError: 356191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788612397 with bad export cookie 3196147887592641486 [11247.626579] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11247.638318] LustreError: 356191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11254.084223] Lustre: server umount lustre-MDT0001 complete [11265.994892] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [11268.973572] Key type lgssc unregistered [11269.267800] LNet: 360572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11269.275208] LNetError: 360572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11269.296966] LNet: Removed LNI 192.168.201.103@tcp [11270.206688] Key type .llcrypt unregistered [11270.208683] Key type ._llcrypt unregistered [11333.844858] Key type ._llcrypt registered [11333.863987] Key type .llcrypt registered [11334.080533] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [11336.428854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11336.485715] alg: No test for adler32 (adler32-zlib) [11337.817374] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [11338.252947] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [11340.111224] Key type lgssc registered [11343.065753] Lustre: Echo OBD driver; http://www.lustre.org/ [11356.926108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11356.967238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11358.743433] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11358.797638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11359.266505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11365.325914] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11376.790725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11376.976180] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11377.021155] Lustre: 363359:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11377.524691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11383.386585] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11390.428518] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11394.722380] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11398.112705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11398.117961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11398.126408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11401.188799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11401.190452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11403.497061] Lustre: server umount lustre-MDT0000 complete [11407.669356] LustreError: 363350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788612557 with bad export cookie 281168821751537117 [11407.670804] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11407.682317] LustreError: 363350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11408.389505] Lustre: server umount lustre-MDT0001 complete [11418.330506] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 08:49:27 (1788612567) [11425.379067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11425.831684] LustreError: 365645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11425.839606] LustreError: 365645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11425.898394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11430.678392] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11431.396368] LustreError: 365646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11436.524262] LustreError: 365645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11440.280839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11440.777000] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11445.022560] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11451.688267] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11455.442631] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11462.684322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11463.004765] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11463.220520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11469.802298] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11476.388164] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11484.787927] Lustre: Permanently deactivating lustre-OST0000 [11484.813989] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11494.449103] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11496.243528] LustreError: 368340:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11496.255599] LustreError: 368340:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [11496.262311] LustreError: 368340:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11499.880878] LustreError: 368389:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [11499.891759] LustreError: 368389:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11508.180863] Lustre: server umount lustre-OST0000 complete [11511.423141] LustreError: 368638:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [11511.431901] LustreError: 368638:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11517.409968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11517.416244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11517.430468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11517.435643] Lustre: Skipped 1 previous similar message [11523.555478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11523.566412] Lustre: Skipped 3 previous similar messages [11524.651608] Lustre: server umount lustre-MDT0000 complete [11528.673316] LustreError: 365641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11528.692326] LustreError: 365641:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11528.906819] LustreError: 365627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788612678 with bad export cookie 281168821751537698 [11528.913304] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11528.936442] LustreError: 365627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11538.315892] Lustre: server umount lustre-MDT0001 complete [11548.026984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11553.016178] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11556.327590] LustreError: 369781:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11563.782784] Lustre: 370120:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.103@tcp [11566.469075] Lustre: server umount MGS complete [11579.680458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11580.153579] LustreError: 370845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11580.184204] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11580.193427] Lustre: Skipped 1 previous similar message [11580.276683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11585.687216] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11590.627487] LustreError: 370846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11590.665509] LustreError: 370846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11596.066777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11596.395052] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11596.526473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11601.667515] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11608.969514] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11613.007539] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11615.157473] Lustre: Permanently reactivating lustre-OST0000 [11615.189813] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11615.198088] Lustre: Skipped 2 previous similar messages [11644.895174] LustreError: 372405:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11653.611584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11654.196694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11662.169660] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11669.025788] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11679.722432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11679.742971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11679.751839] Lustre: Skipped 1 previous similar message [11679.757642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11682.811996] Lustre: server umount lustre-OST0000 complete [11688.419440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11688.431711] LustreError: Skipped 1 previous similar message [11688.443472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11688.476145] Lustre: Skipped 1 previous similar message [11688.488533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11688.495736] Lustre: Skipped 2 previous similar messages [11694.142387] Lustre: server umount lustre-MDT0000 complete [11696.096840] LustreError: 371557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11696.125426] LustreError: 371557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11698.970035] LustreError: 370825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788612849 with bad export cookie 281168821751538748 [11698.987606] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11699.926039] Lustre: server umount lustre-MDT0001 complete [11714.441036] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [11717.415868] Key type lgssc unregistered [11717.674466] LNet: 374857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11717.687809] LNetError: 374857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11717.703150] LNet: Removed LNI 192.168.201.103@tcp [11718.682313] Key type .llcrypt unregistered [11718.692154] Key type ._llcrypt unregistered [11739.235826] Key type ._llcrypt registered [11739.238721] Key type .llcrypt registered [11739.351901] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [11756.394648] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [11758.425725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11758.479484] alg: No test for adler32 (adler32-zlib) [11759.515124] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [11759.771967] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [11761.561938] Key type lgssc registered [11763.098050] Lustre: Echo OBD driver; http://www.lustre.org/ [11801.771049] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11801.797238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11803.231178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11803.288733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11803.404763] Lustre: lustre-MDT0000: new disk, initializing [11803.589524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11803.640205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11810.097418] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11824.598376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11824.761469] Lustre: 377980:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11824.855212] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11824.858267] Lustre: Skipped 1 previous similar message [11824.929496] Lustre: lustre-MDT0001: new disk, initializing [11825.068969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11825.144401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11825.180403] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11830.883335] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11842.191715] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11846.385726] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11855.761624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11856.235854] Lustre: lustre-OST0000: new disk, initializing [11856.243235] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11856.371557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11863.181233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11863.209520] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11863.402527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11864.173904] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11875.516750] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11882.428996] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11882.774996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11888.823863] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11889.106927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11894.249025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11894.249673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11894.264826] Lustre: Skipped 1 previous similar message [11894.266230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11894.269394] LustreError: Skipped 1 previous similar message [11897.234602] Lustre: server umount lustre-OST0000 complete [11901.921784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11901.933343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11901.956880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11901.960537] Lustre: Skipped 1 previous similar message [11904.480983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11904.484330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11904.489101] Lustre: Skipped 1 previous similar message [11908.135985] Lustre: server umount lustre-MDT0000 complete [11912.926552] LustreError: 377973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788613063 with bad export cookie 12646709007408106563 [11912.935166] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11912.946684] LustreError: 377973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11913.654377] Lustre: server umount lustre-MDT0001 complete [11922.431268] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11923.977420] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 08:57:53 (1788613073) [11930.576410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11931.084320] LustreError: 382036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11931.111861] LustreError: 382036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11931.225200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11935.593364] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11936.227056] LustreError: 382037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11941.360937] LustreError: 382036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11945.368297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11945.789311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11950.359742] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11956.679904] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11961.364553] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11969.092059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11969.643070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11976.903637] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11983.517923] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11985.454096] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11987.963579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11996.642427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11996.649718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11996.677471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12000.686573] Lustre: server umount lustre-OST0000 complete [12007.392645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12007.400665] LustreError: Skipped 1 previous similar message [12007.403191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12007.409822] Lustre: Skipped 1 previous similar message [12007.412872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12007.422540] Lustre: Skipped 2 previous similar messages [12012.406055] Lustre: server umount lustre-MDT0000 complete [12013.031970] LustreError: 382037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12018.145831] LustreError: 382032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12018.164752] LustreError: 382032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12018.257075] LustreError: 382754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788613168 with bad export cookie 12646709007408107445 [12018.266234] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12018.272447] LustreError: 382754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12019.119723] Lustre: server umount lustre-MDT0001 complete [12033.774749] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [12036.552530] Key type lgssc unregistered [12037.006565] LNet: 385412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12037.012828] LNetError: 385412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12037.033597] LNet: Removed LNI 192.168.201.103@tcp [12037.876444] Key type .llcrypt unregistered [12037.878427] Key type ._llcrypt unregistered [12064.327701] Key type ._llcrypt registered [12064.332114] Key type .llcrypt registered [12064.528956] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [12064.891676] Key type .llcrypt unregistered [12064.894432] Key type ._llcrypt unregistered [12085.155414] Key type ._llcrypt registered [12085.160470] Key type .llcrypt registered [12085.296455] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [12086.920792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12087.036078] alg: No test for adler32 (adler32-zlib) [12088.131697] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [12088.254425] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [12089.873478] Key type lgssc registered [12091.090458] Lustre: Echo OBD driver; http://www.lustre.org/ [12099.831583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12099.869214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12101.530230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12106.910661] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12111.859401] LustreError: 387595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12111.896670] LustreError: 387595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12116.962781] LustreError: 387596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12116.984490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12117.497409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12122.607118] LustreError: 387590:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12122.645838] LustreError: 387590:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12124.049110] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12126.697853] LustreError: 387595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12126.729530] LustreError: 387595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12126.730930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12129.784040] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12131.815393] LustreError: 387592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12131.834544] LustreError: 387592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12138.041602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12138.747338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12143.247424] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12154.338963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12154.353311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12154.362623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12154.377358] Lustre: Skipped 1 previous similar message [12154.394458] Lustre: Skipped 1 previous similar message [12159.460558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12159.476794] Lustre: Skipped 1 previous similar message [12159.695828] Lustre: server umount lustre-OST0000 complete [12167.145484] Lustre: server umount lustre-MDT0000 complete [12169.184111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12169.192947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12169.208406] LustreError: 387596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12169.235526] LustreError: 387596:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12171.121104] LustreError: 387574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788613321 with bad export cookie 4545661052569869586 [12171.135100] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12171.137174] LustreError: 387574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12171.646954] Lustre: server umount lustre-MDT0001 complete [12186.124833] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [12189.885709] Key type lgssc unregistered [12190.183749] LNet: 390594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12190.190419] LNetError: 390594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12191.206780] LNet: Removed LNI 192.168.201.103@tcp [12192.559257] Key type .llcrypt unregistered [12192.567520] Key type ._llcrypt unregistered [12220.969551] Key type ._llcrypt registered [12220.971377] Key type .llcrypt registered [12221.064914] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [12222.196989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12222.325289] alg: No test for adler32 (adler32-zlib) [12223.483972] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [12223.671124] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [12225.343228] Key type lgssc registered [12226.899301] Lustre: Echo OBD driver; http://www.lustre.org/ [12236.678773] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12236.694252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12252.258564] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12257.379176] LustreError: 391969:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.103@tcp: failed processing log, type 4: rc = -110 [12284.063285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12290.189318] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12296.182475] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12304.558491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12304.961162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12305.044238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12310.000670] LustreError: 391995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12310.023660] LustreError: 391995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12310.325670] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12315.115408] LustreError: 391996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12318.249077] LustreError: 392894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12318.294319] LustreError: 392894:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12321.174945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12321.609407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12327.156720] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12335.157510] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12339.725032] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12347.365124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12347.370902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12347.400443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12349.298304] Lustre: server umount lustre-OST0000 complete [12355.050172] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12355.057349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12355.077320] Lustre: Skipped 1 previous similar message [12355.094119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12355.107841] Lustre: Skipped 1 previous similar message [12357.607770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12357.607903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12357.650032] Lustre: Skipped 1 previous similar message [12359.606413] Lustre: server umount lustre-MDT0000 complete [12362.746830] LustreError: 392895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12362.767683] LustreError: 392895:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12363.866526] LustreError: 392924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788613513 with bad export cookie 8452848982824176567 [12363.870510] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12363.879550] LustreError: 392924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12364.444034] Lustre: server umount lustre-MDT0001 complete [12375.517670] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [12379.329545] Key type lgssc unregistered [12379.685774] LNet: 395190:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12379.700731] LNetError: 395190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12379.745529] LNet: Removed LNI 192.168.201.103@tcp [12380.829863] Key type .llcrypt unregistered [12380.832627] Key type ._llcrypt unregistered [12407.692971] Key type ._llcrypt registered [12407.696254] Key type .llcrypt registered [12407.789795] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [12408.907656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12408.975494] alg: No test for adler32 (adler32-zlib) [12410.156407] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [12410.441326] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [12412.219907] Key type lgssc registered [12413.712786] Lustre: Echo OBD driver; http://www.lustre.org/ [12421.387380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12421.411839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12423.079193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12427.843907] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12433.384313] LustreError: 396591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12433.409684] LustreError: 396591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12436.747890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12437.133156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12441.850396] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12447.910560] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12451.668920] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12458.751236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12459.098657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12465.231985] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12468.345300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12471.285958] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12477.928145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12477.934241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12477.959255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12478.954565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12478.954883] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12478.963310] Lustre: Skipped 1 previous similar message [12478.973761] Lustre: Skipped 1 previous similar message [12482.346806] Lustre: server umount lustre-MDT0000 complete [12484.065166] LustreError: 396591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12484.091493] LustreError: 396591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12484.771454] LustreError: 396573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.3@tcp arrived at 1788613634 with bad export cookie 1338296992491688672 [12486.546854] LustreError: 396571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788613636 with bad export cookie 1338296992491687769 [12486.557866] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12486.560525] LustreError: 396571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12488.161483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12488.175785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12488.206082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12488.214859] Lustre: Skipped 2 previous similar messages [12489.184755] LustreError: 396592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12489.199140] LustreError: 396592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12491.235454] LustreError: 396591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12491.239522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12491.274779] LustreError: 396591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12491.280946] Lustre: Skipped 1 previous similar message [12492.826892] Lustre: server umount lustre-OST0000 complete [12496.352425] LustreError: 396592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12496.379465] LustreError: 396592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12500.139389] Lustre: server umount lustre-MDT0001 complete [12512.381761] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [12515.502805] Key type lgssc unregistered [12515.846623] LNet: 399920:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12515.855940] LNetError: 399920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12515.889945] LNet: Removed LNI 192.168.201.103@tcp [12516.749996] Key type .llcrypt unregistered [12516.752400] Key type ._llcrypt unregistered [12545.958922] Key type ._llcrypt registered [12545.967597] Key type .llcrypt registered [12546.098399] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [12547.268778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12547.319284] alg: No test for adler32 (adler32-zlib) [12548.521465] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [12548.850368] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [12550.631717] Key type lgssc registered [12552.455609] Lustre: Echo OBD driver; http://www.lustre.org/ [12560.069551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12560.088446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12561.917581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12566.876000] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12571.123986] LustreError: 401323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12571.158571] LustreError: 401323:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12576.171403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12576.224607] LustreError: 401324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12576.693326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12581.361407] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12586.398476] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12589.961899] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12596.423470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12596.798951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12602.848247] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12606.009955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12608.113943] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12615.931402] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.3@tcp (stopping) [12616.168102] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12619.599409] Lustre: server umount lustre-MDT0001 complete [12621.004227] LustreError: 401320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12626.406970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12626.410091] LustreError: 401323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12626.417659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12626.417668] Lustre: Skipped 1 previous similar message [12626.420440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12626.441942] LustreError: 401323:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12626.449762] Lustre: Skipped 2 previous similar messages [12631.520571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12631.522989] LustreError: 402298:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12631.685260] Lustre: server umount lustre-OST0000 complete [12635.694275] Lustre: server umount lustre-MDT0000 complete [12651.083253] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [12654.077828] Key type lgssc unregistered [12654.401174] LNet: 404654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12654.414989] LNetError: 404654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12655.464728] LNet: Removed LNI 192.168.201.103@tcp [12656.546515] Key type .llcrypt unregistered [12656.549053] Key type ._llcrypt unregistered [12682.890656] Key type ._llcrypt registered [12682.892680] Key type .llcrypt registered [12683.006486] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [12698.915814] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [12700.569872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12700.749892] alg: No test for adler32 (adler32-zlib) [12701.883682] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [12702.206133] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [12703.920749] Key type lgssc registered [12705.393112] Lustre: Echo OBD driver; http://www.lustre.org/ [12744.641812] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [12760.364477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12760.392873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12761.688496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12761.759158] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12761.935267] Lustre: lustre-MDT0000: new disk, initializing [12762.131186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12762.165794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12766.794080] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12783.389261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12783.774388] Lustre: lustre-OST0000: new disk, initializing [12783.781213] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12783.794258] Lustre: Skipped 1 previous similar message [12783.887637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12790.426191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12790.442591] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12790.492920] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12790.813448] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12798.155619] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12814.565586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12814.707380] Lustre: 408625:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12814.783890] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12814.839817] Lustre: lustre-MDT0001: new disk, initializing [12814.941222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12814.972271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12814.982390] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12818.915788] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12828.019952] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12828.264573] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12832.775032] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12833.050470] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12838.372451] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12838.384423] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12838.397759] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12838.409685] Lustre: Skipped 1 previous similar message [12841.684862] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.3@tcp (stopping) [12843.435514] Lustre: server umount lustre-MDT0001 complete [12848.612196] LustreError: 408637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12848.631644] LustreError: 408637:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [12853.755460] LustreError: 411253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12853.782376] LustreError: 411253:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12858.857166] LustreError: 408631:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12858.886942] LustreError: 408631:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12862.432630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12862.443098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12862.451559] Lustre: Skipped 1 previous similar message [12862.459182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12863.978016] LustreError: 411253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12869.090590] LustreError: 408636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12869.094845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12869.142901] Lustre: Skipped 1 previous similar message [12874.207811] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12874.213456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12874.406817] Lustre: server umount lustre-OST0000 complete [12884.448990] Lustre: server umount lustre-MDT0000 complete [12898.431890] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [12900.966449] Key type lgssc unregistered [12901.315725] LNet: 413197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12901.329579] LNetError: 413197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12901.342985] LNet: Removed LNI 192.168.201.103@tcp [12902.241138] Key type .llcrypt unregistered [12902.245284] Key type ._llcrypt unregistered [12938.779975] Key type ._llcrypt registered [12938.783970] Key type .llcrypt registered [12938.873450] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [12939.100239] Key type .llcrypt unregistered [12939.101565] Key type ._llcrypt unregistered [12954.682363] Key type ._llcrypt registered [12954.690709] Key type .llcrypt registered [12954.846736] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [12955.724438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12955.823522] alg: No test for adler32 (adler32-zlib) [12957.010356] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [12957.281950] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [12958.959434] Key type lgssc registered [12959.900045] Lustre: Echo OBD driver; http://www.lustre.org/ [12988.708293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12988.744744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12990.241546] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12990.324490] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12990.548479] Lustre: lustre-MDT0000: new disk, initializing [12990.824938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12990.863797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12995.890805] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13006.535973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13006.691851] Lustre: 416540:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13006.762888] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13006.771659] Lustre: Skipped 1 previous similar message [13006.832403] Lustre: lustre-MDT0001: new disk, initializing [13006.981656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13007.031408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13007.043980] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13011.250283] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13018.033743] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13021.534904] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13027.409869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13027.616451] Lustre: lustre-OST0000: new disk, initializing [13027.619926] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13027.696427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13032.870371] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13036.654349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13036.672241] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13036.788913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13040.107500] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13047.266602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13047.272647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13047.279911] Lustre: Skipped 1 previous similar message [13047.293871] Lustre: Skipped 2 previous similar messages [13052.032229] Lustre: server umount lustre-MDT0000 complete [13055.417986] LustreError: 418280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788614205 with bad export cookie 5602987564478110421 [13055.423129] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13055.429080] LustreError: 418280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13055.732564] Lustre: server umount lustre-MDT0001 complete [13069.539925] Lustre: server umount lustre-OST0000 complete [13081.410424] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 09:17:10 (1788614230) [13090.246658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13090.786566] LustreError: 420198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13090.802921] LustreError: 420198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13090.846540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13095.913487] LustreError: 420199:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13096.651761] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13100.000797] LustreError: 420198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13105.121378] LustreError: 420199:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13107.129062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13107.630203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13112.596892] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13118.683453] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13122.288620] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13141.998819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13142.323872] Lustre: Found index 0 for lustre-OST0000, updating log [13142.617641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13148.238914] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13153.350982] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13160.055184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13169.632826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13169.654461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13169.663356] Lustre: Skipped 1 previous similar message [13169.680378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13170.720495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13170.727041] Lustre: Skipped 1 previous similar message [13175.385561] Lustre: server umount lustre-OST0000 complete [13192.172707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13192.178245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13192.191778] Lustre: Skipped 2 previous similar messages [13192.195138] Lustre: Skipped 1 previous similar message [13196.320390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13196.332665] Lustre: Skipped 1 previous similar message [13197.912418] Lustre: server umount lustre-MDT0000 complete [13202.418203] LustreError: 420198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13202.467333] LustreError: 420198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13202.573470] LustreError: 420178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788614352 with bad export cookie 5602987564478111807 [13202.594924] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13202.595171] LustreError: 420178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13203.362290] Lustre: server umount lustre-MDT0001 complete [13207.135685] LNet: 1 local NIs in recovery (showing 1): 192.168.201.103@tcp [13218.706240] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 09:19:26 (1788614366) [13836.808643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13837.135557] LustreError: 426297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13837.204744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13841.043459] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13842.404146] LustreError: 426298:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13847.525291] LustreError: 426297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13849.526381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13849.886362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13854.262314] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13860.131676] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13863.795658] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13883.180984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13883.544117] Lustre: Found index 0 for lustre-OST0000, updating log [13883.843704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13885.880483] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [13889.857573] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13895.770769] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13898.399172] LNet: 1 local NIs in recovery (showing 1): 192.168.201.103@tcp [13901.285784] LustreError: lustre-OST0000-osc-MDT0001: operation ost_create to node 0@lo failed: rc = -107 [13901.289370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13901.291426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13901.291434] Lustre: Skipped 3 previous similar messages [13901.301412] LustreError: Skipped 1 previous similar message [13901.331347] Lustre: Skipped 1 previous similar message [13904.572892] Lustre: server umount lustre-OST0000 complete [13909.930554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13909.936028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13909.960514] Lustre: Skipped 1 previous similar message [13909.964418] Lustre: Skipped 1 previous similar message [13912.033220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13912.038229] Lustre: Skipped 4 previous similar messages [13916.130681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13916.139208] Lustre: Skipped 7 previous similar messages [13923.295160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13923.498126] Lustre: server umount lustre-MDT0000 complete [13924.323056] LustreError: 427302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13924.375606] LustreError: 427302:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13927.909734] LustreError: 426280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788615077 with bad export cookie 5602987564478112899 [13927.915905] LustreError: 427009:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [13927.919540] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13927.924768] LustreError: 426280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [13927.987308] LustreError: 427009:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [13928.269221] Lustre: server umount lustre-MDT0001 complete [13959.073911] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [13962.915060] LNet: 1 local NIs in recovery (showing 1): 192.168.201.103@tcp [13968.039716] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [14001.673713] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 09:32:30 (1788615150) [14017.372754] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [14019.668105] Key type lgssc unregistered [14020.017900] LNet: 432519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14020.039222] LNetError: 432519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14145.002881] LNet: Removed LNI 192.168.201.103@tcp [14145.780663] Key type .llcrypt unregistered [14145.784205] Key type ._llcrypt unregistered [14161.837525] Key type ._llcrypt registered [14161.842520] Key type .llcrypt registered [14161.940142] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [14163.160383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14163.177483] alg: No test for adler32 (adler32-zlib) [14164.189894] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [14164.419415] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [14166.087216] Key type lgssc registered [14167.158341] Lustre: Echo OBD driver; http://www.lustre.org/ [14170.997628] LNet: Added LNI 192.168.201.103@tcp100 [8/256/0/180] [14171.848747] LNet: Added LNI 192.168.201.103@tcp101 [8/256/0/180] [14172.683270] LNet: Added LNI 192.168.201.103@tcp102 [8/256/0/180] [14173.497534] LNet: Added LNI 192.168.201.103@tcp103 [8/256/0/180] [14174.314586] LNet: Added LNI 192.168.201.103@tcp104 [8/256/0/180] [14175.109674] LNet: Added LNI 192.168.201.103@tcp105 [8/256/0/180] [14176.159790] LNet: Added LNI 192.168.201.103@tcp106 [8/256/0/180] [14176.876545] LNet: Added LNI 192.168.201.103@tcp107 [8/256/0/180] [14177.639533] LNet: Added LNI 192.168.201.103@tcp108 [8/256/0/180] [14178.526911] LNet: Added LNI 192.168.201.103@tcp109 [8/256/0/180] [14179.458448] LNet: Added LNI 192.168.201.103@tcp110 [8/256/0/180] [14180.473656] LNet: Added LNI 192.168.201.103@tcp111 [8/256/0/180] [14181.211816] LNet: Added LNI 192.168.201.103@tcp112 [8/256/0/180] [14182.168729] LNet: Added LNI 192.168.201.103@tcp113 [8/256/0/180] [14183.045749] LNet: Added LNI 192.168.201.103@tcp114 [8/256/0/180] [14183.762500] LNet: Added LNI 192.168.201.103@tcp115 [8/256/0/180] [14184.634852] LNet: Added LNI 192.168.201.103@tcp116 [8/256/0/180] [14185.352315] LNet: Added LNI 192.168.201.103@tcp117 [8/256/0/180] [14186.272880] LNet: Added LNI 192.168.201.103@tcp118 [8/256/0/180] [14187.275854] LNet: Added LNI 192.168.201.103@tcp119 [8/256/0/180] [14188.246852] LNet: Added LNI 192.168.201.103@tcp120 [8/256/0/180] [14189.506739] LNet: Added LNI 192.168.201.103@tcp121 [8/256/0/180] [14190.607980] LNet: Added LNI 192.168.201.103@tcp122 [8/256/0/180] [14191.565046] LNet: Added LNI 192.168.201.103@tcp123 [8/256/0/180] [14192.413764] LNet: Added LNI 192.168.201.103@tcp124 [8/256/0/180] [14193.424718] LNet: Added LNI 192.168.201.103@tcp125 [8/256/0/180] [14194.220356] LNet: Added LNI 192.168.201.103@tcp126 [8/256/0/180] [14195.049114] LNet: Added LNI 192.168.201.103@tcp127 [8/256/0/180] [14196.047550] LNet: Added LNI 192.168.201.103@tcp128 [8/256/0/180] [14196.944380] LNet: Added LNI 192.168.201.103@tcp129 [8/256/0/180] [14197.881954] LNet: Added LNI 192.168.201.103@tcp130 [8/256/0/180] [14198.893287] LNet: Added LNI 192.168.201.103@tcp131 [8/256/0/180] [14199.880714] LNet: Added LNI 192.168.201.103@tcp132 [8/256/0/180] [14200.855930] LNet: Added LNI 192.168.201.103@tcp133 [8/256/0/180] [14201.714198] LNet: Added LNI 192.168.201.103@tcp134 [8/256/0/180] [14202.726769] LNet: Added LNI 192.168.201.103@tcp135 [8/256/0/180] [14209.162761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14209.242245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14210.367355] Lustre: 435533:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.103@tcp: can't add NID 192.168.201.103@tcp130: rc = -75 [14210.452143] LNetError: 435556:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [14210.468296] LNetError: Protocol error connecting to 192.168.201.103@tcp122 at host 192.168.201.103:988: is it running a compatible version of Lustre? [14210.494375] LNetError: Skipped 10 previous similar messages [14210.527217] LNet: 8 local NIs in recovery (showing 8): 192.168.201.103@tcp118, 192.168.201.103@tcp123, 192.168.201.103@tcp125, 192.168.201.103@tcp127, 192.168.201.103@tcp124, 192.168.201.103@tcp129, 192.168.201.103@tcp128, 192.168.201.103@tcp122 [14210.582198] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14210.618032] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14210.715648] Lustre: lustre-MDT0000: new disk, initializing [14210.831972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14210.874122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14211.493230] LNetError: 432992:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [14211.502087] LNetError: 432992:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 10 previous similar messages [14211.514567] LNetError: Protocol error connecting to 192.168.201.103@tcp111 at host 192.168.201.103:988: is it running a compatible version of Lustre? [14213.667325] LNetError: 435556:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [14213.680424] LNetError: 435556:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 8 previous similar messages [14213.700975] LNetError: Protocol error connecting to 192.168.201.103@tcp118 at host 192.168.201.103:988: is it running a compatible version of Lustre? [14213.715233] LNetError: Skipped 11 previous similar messages [14215.941299] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [14217.761813] LNetError: 435553:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [14217.772897] LNetError: 435553:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 5 previous similar messages [14217.781235] LNetError: Protocol error connecting to 192.168.201.103@tcp118 at host 192.168.201.103:988: is it running a compatible version of Lustre? [14217.794753] LNetError: Skipped 2 previous similar messages [14225.952622] LNetError: 435566:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [14225.957954] LNetError: Protocol error connecting to 192.168.201.103@tcp118 at host 192.168.201.103:988: is it running a compatible version of Lustre? [14227.477341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14227.586563] Lustre: 435551:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14227.659688] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14227.665639] Lustre: Skipped 1 previous similar message [14227.777645] Lustre: lustre-MDT0001: new disk, initializing [14227.820461] Lustre: 436335:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.103@tcp: can't add NID 192.168.201.103@tcp130: rc = -75 [14227.829912] Lustre: 436335:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [14227.939995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14227.994983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14228.028660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14232.797717] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [14237.206644] Lustre: 436743:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.103@tcp: can't add NID 192.168.201.103@tcp130: rc = -75 [14237.210910] Lustre: 436743:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [14240.816440] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14241.316083] LNetError: 435559:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [14241.335265] LNetError: Protocol error connecting to 192.168.201.103@tcp118 at host 192.168.201.103:988: is it running a compatible version of Lustre? [14244.895488] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14251.194602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14251.604930] Lustre: lustre-OST0000: new disk, initializing [14251.610327] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14251.721810] Lustre: 437304:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.103@tcp: can't add NID 192.168.201.103@tcp130: rc = -75 [14251.739306] Lustre: 437304:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [14251.839414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14257.197320] Lustre: 437593:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.103@tcp: can't add NID 192.168.201.103@tcp130: rc = -75 [14257.202993] Lustre: 437593:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [14257.276633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14257.287200] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14257.422327] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14257.630619] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [14257.635757] LNetError: 435560:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [14257.648071] LNetError: 435560:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 10 previous similar messages [14257.657394] LNetError: Protocol error connecting to 192.168.201.103@tcp111 at host 192.168.201.103:988: is it running a compatible version of Lustre? [14257.675248] LNetError: Skipped 10 previous similar messages [14264.560290] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14269.983192] LNet: 1 local NIs in recovery (showing 1): 192.168.201.103@tcp118 [14272.994816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14272.996630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14273.012567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14273.045103] Lustre: Skipped 1 previous similar message [14277.345567] Lustre: server umount lustre-OST0000 complete [14283.237445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14283.243642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14283.264969] Lustre: Skipped 2 previous similar messages [14287.800738] Lustre: server umount lustre-MDT0000 complete [14290.407277] LNetError: 432993:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [14290.426385] LNetError: 432993:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 20 previous similar messages [14290.437308] LNetError: Protocol error connecting to 192.168.201.103@tcp128 at host 192.168.201.103:988: is it running a compatible version of Lustre? [14290.441942] LNetError: Skipped 20 previous similar messages [14291.925782] LustreError: 435542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788615442 with bad export cookie 5270390641986360816 [14291.929994] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14291.940694] LustreError: 435542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14292.336478] Lustre: server umount lustre-MDT0001 complete [14303.211911] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [14306.072350] Key type lgssc unregistered [14306.639115] LNet: 439083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14306.650275] LNetError: 439083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14307.679178] LNet: Removed LNI 192.168.201.103@tcp [14307.703625] LNetError: 439083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14307.726333] LNet: Removed LNI 192.168.201.103@tcp100 [14308.767544] LNet: Removed LNI 192.168.201.103@tcp101 [14308.769595] LNetError: 439083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14308.778388] LNetError: 439083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14308.786265] LNet: Removed LNI 192.168.201.103@tcp102 [14309.791161] LNet: Removed LNI 192.168.201.103@tcp103 [14310.821857] LNet: Removed LNI 192.168.201.103@tcp104 [14310.829855] LNetError: 439083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14310.845171] LNetError: 439083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14310.862490] LNet: Removed LNI 192.168.201.103@tcp105 [14311.904462] LNet: Removed LNI 192.168.201.103@tcp106 [14312.927219] LNet: Removed LNI 192.168.201.103@tcp107 [14313.951152] LNet: Removed LNI 192.168.201.103@tcp108 [14314.975456] LNet: Removed LNI 192.168.201.103@tcp109 [14314.978573] LNetError: 439083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14314.987057] LNetError: 439083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14315.015335] LNet: Removed LNI 192.168.201.103@tcp110 [14316.063147] LNet: Removed LNI 192.168.201.103@tcp111 [14317.087182] LNet: Removed LNI 192.168.201.103@tcp112 [14318.111372] LNet: Removed LNI 192.168.201.103@tcp113 [14319.137197] LNet: Removed LNI 192.168.201.103@tcp114 [14320.160758] LNet: Removed LNI 192.168.201.103@tcp115 [14321.183363] LNet: Removed LNI 192.168.201.103@tcp116 [14322.207195] LNet: Removed LNI 192.168.201.103@tcp117 [14322.210358] LNet: Removed LNI 192.168.201.103@tcp118 [14323.231156] LNet: Removed LNI 192.168.201.103@tcp119 [14323.237965] LNetError: 439083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14323.251515] LNetError: 439083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [14323.270090] LNet: Removed LNI 192.168.201.103@tcp120 [14324.319449] LNet: Removed LNI 192.168.201.103@tcp121 [14325.343520] LNet: Removed LNI 192.168.201.103@tcp122 [14326.367285] LNet: Removed LNI 192.168.201.103@tcp123 [14327.391182] LNet: Removed LNI 192.168.201.103@tcp124 [14328.416904] LNet: Removed LNI 192.168.201.103@tcp125 [14329.439625] LNet: Removed LNI 192.168.201.103@tcp126 [14330.463183] LNet: Removed LNI 192.168.201.103@tcp127 [14331.487366] LNet: Removed LNI 192.168.201.103@tcp128 [14332.511155] LNet: Removed LNI 192.168.201.103@tcp129 [14332.517940] LNet: Removed LNI 192.168.201.103@tcp130 [14332.520672] LNet: Removed LNI 192.168.201.103@tcp131 [14332.526906] LNet: Removed LNI 192.168.201.103@tcp132 [14332.537116] LNet: Removed LNI 192.168.201.103@tcp133 [14332.544943] LNet: Removed LNI 192.168.201.103@tcp134 [14332.565229] LNet: Removed LNI 192.168.201.103@tcp135 [14333.574277] Key type .llcrypt unregistered [14333.577824] Key type ._llcrypt unregistered [14356.311903] Key type ._llcrypt registered [14356.314907] Key type .llcrypt registered [14356.422251] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [14371.142823] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [14372.221810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14372.375417] alg: No test for adler32 (adler32-zlib) [14373.516583] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [14373.809122] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [14375.503546] Key type lgssc registered [14376.658686] Lustre: Echo OBD driver; http://www.lustre.org/ [14411.581444] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 09:39:20 (1788615560) [14431.342485] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [14435.150676] Key type lgssc unregistered [14435.526268] LNet: 443121:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14435.549260] LNetError: 443121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14435.596796] LNet: Removed LNI 192.168.201.103@tcp [14436.768957] Key type .llcrypt unregistered [14436.772236] Key type ._llcrypt unregistered [14454.933703] Key type ._llcrypt registered [14454.935440] Key type .llcrypt registered [14455.069911] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [14456.632600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14456.665639] alg: No test for adler32 (adler32-zlib) [14457.760329] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [14458.110483] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [14459.856347] Key type lgssc registered [14461.061951] Lustre: Echo OBD driver; http://www.lustre.org/ [14470.075795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14470.094700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14471.430664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14471.472481] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14471.591877] Lustre: lustre-MDT0000: new disk, initializing [14471.668133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14471.684562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14476.155371] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [14485.382356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14485.499436] Lustre: 444316:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14485.570928] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14485.577635] Lustre: Skipped 1 previous similar message [14485.646382] Lustre: lustre-MDT0001: new disk, initializing [14485.740767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14485.762935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14485.772696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14489.248069] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [14496.773863] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14500.994805] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14507.670379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14507.894518] Lustre: lustre-OST0000: new disk, initializing [14507.897919] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14507.971625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14514.288649] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [14516.288883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14516.301265] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14516.429558] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14521.293090] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14524.926680] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing lnet_if_list [14529.646681] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14529.667979] LNet: Added LNI 192.168.201.103@tcp10 [8/256/0/180] [14535.334367] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14535.360954] LNet: Added LNI 192.168.201.103@tcp11 [8/256/0/180] [14538.918558] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14538.947975] LNet: Added LNI 192.168.201.103@tcp12 [8/256/0/180] [14542.596505] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14542.623265] LNet: Added LNI 192.168.201.103@tcp13 [8/256/0/180] [14547.742035] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14547.796120] LNet: Added LNI 192.168.201.103@tcp14 [8/256/0/180] [14551.757501] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14551.781039] LNet: Added LNI 192.168.201.103@tcp15 [8/256/0/180] [14555.308886] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14555.333509] LNet: Added LNI 192.168.201.103@tcp16 [8/256/0/180] [14559.375472] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14559.403947] LNet: Added LNI 192.168.201.103@tcp17 [8/256/0/180] [14564.550359] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14564.597250] LNet: Added LNI 192.168.201.103@tcp18 [8/256/0/180] [14569.707350] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14569.730391] LNet: Added LNI 192.168.201.103@tcp19 [8/256/0/180] [14574.452721] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14574.497961] LNet: Added LNI 192.168.201.103@tcp20 [8/256/0/180] [14580.606507] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14580.652172] LNet: Added LNI 192.168.201.103@tcp21 [8/256/0/180] [14587.134243] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14587.157771] LNet: Added LNI 192.168.201.103@tcp22 [8/256/0/180] [14592.905219] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14592.929639] LNet: Added LNI 192.168.201.103@tcp23 [8/256/0/180] [14597.555389] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14597.579634] LNet: Added LNI 192.168.201.103@tcp24 [8/256/0/180] [14602.806995] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14602.839715] LNet: Added LNI 192.168.201.103@tcp25 [8/256/0/180] [14608.970406] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14609.013163] LNet: Added LNI 192.168.201.103@tcp26 [8/256/0/180] [14616.417454] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14616.444870] LNet: Added LNI 192.168.201.103@tcp27 [8/256/0/180] [14622.953506] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14622.981604] LNet: Added LNI 192.168.201.103@tcp28 [8/256/0/180] [14629.130290] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14629.205415] LNet: Added LNI 192.168.201.103@tcp29 [8/256/0/180] [14636.549522] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14636.604390] LNet: Added LNI 192.168.201.103@tcp30 [8/256/0/180] [14644.478660] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14644.534616] LNet: Added LNI 192.168.201.103@tcp31 [8/256/0/180] [14652.688383] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14652.719382] LNet: Added LNI 192.168.201.103@tcp32 [8/256/0/180] [14659.779884] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14659.819936] LNet: Added LNI 192.168.201.103@tcp33 [8/256/0/180] [14666.877974] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14666.915323] LNet: Added LNI 192.168.201.103@tcp34 [8/256/0/180] [14673.076828] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14673.104192] LNet: Added LNI 192.168.201.103@tcp35 [8/256/0/180] [14679.409337] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14679.449770] LNet: Added LNI 192.168.201.103@tcp36 [8/256/0/180] [14686.378665] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14686.412283] LNet: Added LNI 192.168.201.103@tcp37 [8/256/0/180] [14692.501473] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14692.527418] LNet: Added LNI 192.168.201.103@tcp38 [8/256/0/180] [14698.917918] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14698.980098] LNet: Added LNI 192.168.201.103@tcp39 [8/256/0/180] [14705.183358] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14705.223248] LNet: Added LNI 192.168.201.103@tcp40 [8/256/0/180] [14712.529243] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14712.555223] LNet: Added LNI 192.168.201.103@tcp41 [8/256/0/180] [14719.640808] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14719.670689] LNet: Added LNI 192.168.201.103@tcp42 [8/256/0/180] [14726.710197] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14726.749386] LNet: Added LNI 192.168.201.103@tcp43 [8/256/0/180] [14734.291580] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14734.319863] LNet: Added LNI 192.168.201.103@tcp44 [8/256/0/180] [14741.218804] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14741.239418] LNet: Added LNI 192.168.201.103@tcp45 [8/256/0/180] [14749.108451] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14749.134241] LNet: Added LNI 192.168.201.103@tcp46 [8/256/0/180] [14757.840371] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14757.866468] LNet: Added LNI 192.168.201.103@tcp47 [8/256/0/180] [14766.584855] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14766.619283] LNet: Added LNI 192.168.201.103@tcp48 [8/256/0/180] [14775.457167] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14775.482667] LNet: Added LNI 192.168.201.103@tcp49 [8/256/0/180] [14785.076593] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14785.116891] LNet: Added LNI 192.168.201.103@tcp50 [8/256/0/180] [14808.544991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14808.547091] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14808.564046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14808.594446] Lustre: Skipped 1 previous similar message [14810.560816] Lustre: server umount lustre-MDT0000 complete [14814.493913] LustreError: 444308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788615964 with bad export cookie 2685478655369683581 [14814.497033] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14814.502439] LustreError: 444308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14814.778631] Lustre: server umount lustre-MDT0001 complete [14832.776906] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [14834.150317] Lustre: 443591:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788615968/real 1788615968] req@000000003ab37683 x1875499399936512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788615984 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14834.188350] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14834.204132] Lustre: Skipped 1 previous similar message [14835.174697] Lustre: 443588:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788615969/real 1788615969] req@00000000864c8b04 x1875499399936640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788615985 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14839.341308] Lustre: 443588:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788615973/real 1788615973] req@00000000c37082ea x1875499399936768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788615989 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14844.993370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14846.434048] Lustre: 443590:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788615980/real 1788615980] req@0000000012bc3a5a x1875499399937152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788615996 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14846.479197] Lustre: 443590:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14861.308809] LustreError: 446054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14861.323945] LustreError: 446054:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [14861.387129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14863.408188] LustreError: 446055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14863.436813] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14864.418970] LustreError: 446054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14867.286355] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [14869.479545] LustreError: 446055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14874.596516] LustreError: 446054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14876.623453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14876.997289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14879.015676] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14881.658975] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [14886.981871] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14890.207862] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14904.805475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14904.811397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14904.818600] Lustre: Skipped 1 previous similar message [14904.825813] Lustre: Skipped 3 previous similar messages [14909.924869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14909.936304] Lustre: Skipped 1 previous similar message [14910.733834] Lustre: server umount lustre-OST0000 complete [14915.040394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14915.046667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14915.049967] Lustre: Skipped 1 previous similar message [14920.161460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14920.172384] Lustre: Skipped 2 previous similar messages [14928.351323] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14928.609963] Lustre: server umount lustre-MDT0000 complete [14930.400405] LustreError: 469576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14930.409868] LustreError: 469576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14931.508539] LustreError: 444310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788616081 with bad export cookie 2685478655369686108 [14931.510294] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14931.516394] LustreError: 444310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14931.759560] Lustre: server umount lustre-MDT0001 complete [14941.394824] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [14944.484534] Key type lgssc unregistered [14944.941770] LNet: 472115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14944.957320] LNetError: 472115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14944.963487] LNet: Removed LNI 192.168.201.103@tcp [14944.967674] LNet: Removed LNI 192.168.201.103@tcp10 [14944.976482] LNet: Removed LNI 192.168.201.103@tcp11 [14944.980474] LNet: Removed LNI 192.168.201.103@tcp12 [14944.983565] LNet: Removed LNI 192.168.201.103@tcp13 [14944.986380] LNet: Removed LNI 192.168.201.103@tcp14 [14944.989473] LNet: Removed LNI 192.168.201.103@tcp15 [14944.996858] LNet: Removed LNI 192.168.201.103@tcp16 [14944.999338] LNet: Removed LNI 192.168.201.103@tcp17 [14945.010299] LNet: Removed LNI 192.168.201.103@tcp18 [14945.015295] LNet: Removed LNI 192.168.201.103@tcp19 [14945.018863] LNet: Removed LNI 192.168.201.103@tcp20 [14945.021276] LNet: Removed LNI 192.168.201.103@tcp21 [14945.028181] LNet: Removed LNI 192.168.201.103@tcp22 [14945.029482] LNet: Removed LNI 192.168.201.103@tcp23 [14945.034726] LNet: Removed LNI 192.168.201.103@tcp24 [14945.038397] LNet: Removed LNI 192.168.201.103@tcp25 [14945.040685] LNet: Removed LNI 192.168.201.103@tcp26 [14945.044333] LNet: Removed LNI 192.168.201.103@tcp27 [14945.049782] LNet: Removed LNI 192.168.201.103@tcp28 [14945.057080] LNet: Removed LNI 192.168.201.103@tcp29 [14945.060809] LNet: Removed LNI 192.168.201.103@tcp30 [14945.063138] LNet: Removed LNI 192.168.201.103@tcp31 [14945.065724] LNet: Removed LNI 192.168.201.103@tcp32 [14945.069409] LNet: Removed LNI 192.168.201.103@tcp33 [14945.076421] LNet: Removed LNI 192.168.201.103@tcp34 [14945.089320] LNet: Removed LNI 192.168.201.103@tcp35 [14945.092470] LNet: Removed LNI 192.168.201.103@tcp36 [14945.094438] LNet: Removed LNI 192.168.201.103@tcp37 [14945.097481] LNet: Removed LNI 192.168.201.103@tcp38 [14945.100336] LNet: Removed LNI 192.168.201.103@tcp39 [14945.103481] LNet: Removed LNI 192.168.201.103@tcp40 [14945.106406] LNet: Removed LNI 192.168.201.103@tcp41 [14945.109383] LNet: Removed LNI 192.168.201.103@tcp42 [14945.112408] LNet: Removed LNI 192.168.201.103@tcp43 [14945.114386] LNet: Removed LNI 192.168.201.103@tcp44 [14945.116994] LNet: Removed LNI 192.168.201.103@tcp45 [14945.119306] LNet: Removed LNI 192.168.201.103@tcp46 [14945.122497] LNet: Removed LNI 192.168.201.103@tcp47 [14945.125471] LNet: Removed LNI 192.168.201.103@tcp48 [14945.131234] LNet: Removed LNI 192.168.201.103@tcp49 [14945.146145] LNet: Removed LNI 192.168.201.103@tcp50 [14945.989126] Key type .llcrypt unregistered [14946.001143] Key type ._llcrypt unregistered [14962.889758] Key type ._llcrypt registered [14962.897547] Key type .llcrypt registered [14963.003728] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [14963.837457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14963.866895] alg: No test for adler32 (adler32-zlib) [14965.170532] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [14965.413863] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [14967.136124] Key type lgssc registered [14968.491401] Lustre: Echo OBD driver; http://www.lustre.org/ [14976.134591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14976.174487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14977.925413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14983.574923] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [14987.242729] LustreError: 473340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14987.276929] LustreError: 473340:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [14992.363544] LustreError: 473339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14994.358230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14994.823710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14999.239518] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [15005.743422] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15010.340585] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15017.935956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15018.374159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15022.448027] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [15024.808706] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [15030.277300] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15041.094449] loop0: detected capacity change from 0 to 204800000 [15041.306234] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15041.342842] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15041.380753] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15047.567419] loop0: detected capacity change from 0 to 204800000 [15047.870698] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15047.888951] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15047.900404] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15053.262711] loop0: detected capacity change from 0 to 204800000 [15058.886297] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15059.036496] Lustre: fs15246-OST0000: new disk, initializing [15059.043614] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15059.116133] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15065.224782] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [15075.257694] loop1: detected capacity change from 0 to 204800000 [15080.768685] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15080.861082] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15080.926978] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15081.014891] Lustre: fs15246-MDT0000: new disk, initializing [15081.153619] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15081.176616] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15081.272279] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15082.341113] Lustre: 477703:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000bdaa14d2 x1875499931449984/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788616248 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15085.403803] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [15087.401851] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15087.418776] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15087.469762] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15111.659807] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15111.680162] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15114.884303] Lustre: server umount fs15246-MDT0000 complete [15116.774479] LustreError: 475010:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15121.892900] LustreError: 473334:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15125.122877] Lustre: server umount fs15246-OST0000 complete [15137.250828] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15137.253915] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15137.289796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15137.299548] Lustre: Skipped 1 previous similar message [15142.329425] Lustre: server umount lustre-OST0000 complete [15147.490787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15147.496656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15147.518063] Lustre: Skipped 2 previous similar messages [15147.540685] Lustre: Skipped 1 previous similar message [15152.608582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15152.612728] Lustre: Skipped 1 previous similar message [15157.730955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15157.743147] Lustre: Skipped 1 previous similar message [15160.287148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15160.506091] Lustre: server umount lustre-MDT0000 complete [15162.852044] LustreError: 474313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15162.862280] LustreError: 474313:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15163.923590] LustreError: 479746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788616314 with bad export cookie 7969918763922405806 [15163.933641] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15163.940855] LustreError: 479746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15164.262967] Lustre: server umount lustre-MDT0001 complete [15175.117708] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [15178.013534] Key type lgssc unregistered [15178.366677] LNet: 480772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15178.376370] LNetError: 480772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15178.394332] LNet: Removed LNI 192.168.201.103@tcp [15179.394857] Key type .llcrypt unregistered [15179.397538] Key type ._llcrypt unregistered [15196.642980] Key type ._llcrypt registered [15196.645413] Key type .llcrypt registered [15196.739443] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [15197.480802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15197.583610] alg: No test for adler32 (adler32-zlib) [15198.708889] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [15198.910035] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [15200.559191] Key type lgssc registered [15201.540771] Lustre: Echo OBD driver; http://www.lustre.org/ [15239.205181] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [15247.600632] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [15281.783474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15281.816436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15283.087232] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15283.141764] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15283.229743] Lustre: lustre-MDT0000: new disk, initializing [15283.339053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15283.359277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15287.304610] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [15298.187854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15298.362458] Lustre: 485186:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15298.465273] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15298.470845] Lustre: Skipped 1 previous similar message [15298.526402] Lustre: lustre-MDT0001: new disk, initializing [15298.589983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15298.623451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15298.636699] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15303.161201] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [15310.102911] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15314.063782] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15322.142526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15322.687281] Lustre: lustre-OST0000: new disk, initializing [15322.693340] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15322.814275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15328.336531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15328.344507] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15328.479356] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15329.856208] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [15337.749688] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15342.208673] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15342.483231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15347.155817] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15347.326145] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15349.216241] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15349.228532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15349.232873] LustreError: Skipped 1 previous similar message [15349.238095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15349.250483] Lustre: Skipped 1 previous similar message [15349.263949] Lustre: Skipped 1 previous similar message [15354.353717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15354.364093] Lustre: Skipped 1 previous similar message [15355.594710] Lustre: server umount lustre-OST0000 complete [15359.463882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15359.466583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15359.480590] Lustre: Skipped 1 previous similar message [15359.507504] Lustre: Skipped 1 previous similar message [15364.577093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15364.592831] Lustre: Skipped 1 previous similar message [15365.624691] Lustre: server umount lustre-MDT0000 complete [15369.217977] LustreError: 485178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788616519 with bad export cookie 8708572415804491893 [15369.220175] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15369.227061] LustreError: 485178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15369.503419] Lustre: server umount lustre-MDT0001 complete [15394.207420] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [15404.479402] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [15439.385609] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15111 sec ========= 09:56:27 (1788616587) [15441.145381] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:56:30 (1788616590) === [15444.336904] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:56:33 (1788616593) === [15485.563061] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [15488.275486] Key type lgssc unregistered [15488.571945] LNet: 493370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15488.575535] LNetError: 493370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15488.588079] LNet: Removed LNI 192.168.201.103@tcp [15489.259529] Key type .llcrypt unregistered [15489.261346] Key type ._llcrypt unregistered