[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 527412663 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.003212] x2apic enabled [ 0.004012] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008777] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010018] pid_max: default: 32768 minimum: 301 [ 0.012035] LSM: Security Framework initializing [ 0.013075] Yama: becoming mindful. [ 0.014000] SELinux: Initializing. [ 0.014079] *** VALIDATE selinux *** [ 0.023409] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028012] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029168] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031134] *** VALIDATE tmpfs *** [ 0.033288] *** VALIDATE proc *** [ 0.034280] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036295] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037165] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040018] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.044748] debug: unmapping init [mem 0xffffffffab259000-0xffffffffab260fff] [ 0.046180] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047892] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048026] ... version: 2 [ 0.049016] ... bit width: 48 [ 0.050016] ... generic registers: 4 [ 0.051012] ... value mask: 0000ffffffffffff [ 0.052016] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054014] ... event mask: 000000070000000f [ 0.055451] rcu: Hierarchical SRCU implementation. [ 0.057694] smp: Bringing up secondary CPUs ... [ 0.058610] x86: Booting SMP configuration: [ 0.059030] .... node #0, CPUs: #1 #2 #3 [ 0.063270] smp: Brought up 1 node, 4 CPUs [ 0.065019] smpboot: Max logical packages: 1 [ 0.066018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.193950] node 0 deferred pages initialised in 124ms [ 0.198199] devtmpfs: initialized [ 0.200432] x86/mm: Memory block size: 128MB [ 0.204800] gcov: version magic: 0x41383552 [ 0.208364] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.209131] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.212552] pinctrl core: initialized pinctrl subsystem [ 0.215198] [ 0.215808] ************************************************************* [ 0.218016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.222017] ** ** [ 0.225013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.227013] ** ** [ 0.230016] ** This means that this kernel is built to expose internal ** [ 0.232011] ** IOMMU data structures, which may compromise security on ** [ 0.234014] ** your system. ** [ 0.237014] ** ** [ 0.240013] ** If you see this message and you are not debugging the ** [ 0.242012] ** kernel, report this immediately to your vendor! ** [ 0.244009] ** ** [ 0.245000] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.245000] ************************************************************* [ 0.247767] NET: Registered protocol family 16 [ 0.249584] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.252059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.255071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.259102] cpuidle: using governor menu [ 0.260774] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.261651] PCI: Using configuration type 1 for base access [ 0.263062] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.274165] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.276021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.282145] cryptd: max_cpu_qlen set to 1000 [ 0.284442] ACPI: Added _OSI(Module Device) [ 0.285030] ACPI: Added _OSI(Processor Device) [ 0.286066] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.288025] ACPI: Added _OSI(Processor Aggregator Device) [ 0.294750] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.303260] ACPI: Interpreter enabled [ 0.304113] ACPI: PM: (supports S0 S3 S4 S5) [ 0.307013] ACPI: Using IOAPIC for interrupt routing [ 0.309122] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.313928] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.326899] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.329043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.332020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.335073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.340482] acpiphp: Slot [2] registered [ 0.341107] acpiphp: Slot [5] registered [ 0.342115] acpiphp: Slot [6] registered [ 0.343183] acpiphp: Slot [7] registered [ 0.345107] acpiphp: Slot [8] registered [ 0.346122] acpiphp: Slot [9] registered [ 0.348135] acpiphp: Slot [10] registered [ 0.349104] acpiphp: Slot [3] registered [ 0.350186] acpiphp: Slot [4] registered [ 0.351130] acpiphp: Slot [11] registered [ 0.352099] acpiphp: Slot [12] registered [ 0.353078] acpiphp: Slot [13] registered [ 0.355100] acpiphp: Slot [14] registered [ 0.356102] acpiphp: Slot [15] registered [ 0.357095] acpiphp: Slot [16] registered [ 0.358107] acpiphp: Slot [17] registered [ 0.359090] acpiphp: Slot [18] registered [ 0.360098] acpiphp: Slot [19] registered [ 0.361122] acpiphp: Slot [20] registered [ 0.362129] acpiphp: Slot [21] registered [ 0.364108] acpiphp: Slot [22] registered [ 0.365119] acpiphp: Slot [23] registered [ 0.367129] acpiphp: Slot [24] registered [ 0.368125] acpiphp: Slot [25] registered [ 0.370127] acpiphp: Slot [26] registered [ 0.371107] acpiphp: Slot [27] registered [ 0.372081] acpiphp: Slot [28] registered [ 0.373102] acpiphp: Slot [29] registered [ 0.374122] acpiphp: Slot [30] registered [ 0.376113] acpiphp: Slot [31] registered [ 0.378086] PCI host bridge to bus 0000:00 [ 0.380025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.382031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.385031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.389050] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.393032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.396034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.399185] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.403616] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.408147] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.420858] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.427064] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.430021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.433021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.436023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.439406] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.443278] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.446191] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.449994] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.457018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.470018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.479018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.484674] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.495016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.506024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.528023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.541958] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.553016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.567017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.603018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.620014] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.632020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.641013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.669036] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.681677] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.690024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.708032] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.734021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.746464] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.754024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.762024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.792021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.807643] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.817018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.824047] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.841014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.851908] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.854554] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.857499] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.860421] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.863355] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.870346] iommu: Default domain type: Passthrough [ 0.871523] SCSI subsystem initialized [ 0.872000] ACPI: bus type USB registered [ 0.874150] usbcore: registered new interface driver usbfs [ 0.877112] usbcore: registered new interface driver hub [ 0.879176] usbcore: registered new device driver usb [ 0.881200] pps_core: LinuxPPS API ver. 1 registered [ 0.883013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.888060] PTP clock support registered [ 0.891227] EDAC MC: Ver: 3.0.0 [ 0.893129] PCI: Using ACPI for IRQ routing [ 0.896449] NetLabel: Initializing [ 0.898019] NetLabel: domain hash size = 128 [ 0.900018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.903104] NetLabel: unlabeled traffic allowed by default [ 0.906188] vgaarb: loaded [ 0.908342] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.911018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.916424] clocksource: Switched to clocksource kvm-clock [ 1.029897] VFS: Disk quotas dquot_6.6.0 [ 1.031449] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.035430] *** VALIDATE ramfs *** [ 1.037148] *** VALIDATE hugetlbfs *** [ 1.038977] pnp: PnP ACPI init [ 1.042341] pnp: PnP ACPI: found 6 devices [ 1.058667] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.062494] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.064776] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.067349] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.069836] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.072631] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.075127] NET: Registered protocol family 2 [ 1.077452] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.082718] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.086915] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.092370] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.099319] TCP: Hash tables configured (established 65536 bind 65536) [ 1.104764] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.109609] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.114259] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.118801] NET: Registered protocol family 1 [ 1.122666] RPC: Registered named UNIX socket transport module. [ 1.125259] RPC: Registered udp transport module. [ 1.127164] RPC: Registered tcp transport module. [ 1.129562] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.133338] NET: Registered protocol family 44 [ 1.135496] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.138719] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.141802] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.145568] PCI: CLS 0 bytes, default 64 [ 1.147598] Unpacking initramfs... [ 2.594953] debug: unmapping init [mem 0xffff97473cc54000-0xffff97473ffbffff] [ 2.599474] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.601913] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.604927] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.113075] Initialise system trusted keyrings [ 3.114467] Key type blacklist registered [ 3.116313] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.125137] zbud: loaded [ 3.130602] *** VALIDATE nfs *** [ 3.131727] *** VALIDATE nfs4 *** [ 3.133285] pstore: using deflate compression [ 3.136592] Platform Keyring initialized [ 3.243560] NET: Registered protocol family 38 [ 3.246486] Key type asymmetric registered [ 3.249152] Asymmetric key parser 'x509' registered [ 3.251339] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.255806] io scheduler mq-deadline registered [ 3.257869] io scheduler kyber registered [ 3.259641] io scheduler bfq registered [ 3.261929] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.265080] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.267702] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.270500] ACPI: Power Button [PWRF] [ 3.276351] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.283616] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.303490] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.316714] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.336091] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.364019] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.394963] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.399880] Non-volatile memory driver v1.3 [ 3.401755] Linux agpgart interface v0.103 [ 3.434122] virtio_blk virtio1: [vda] 146864 512-byte logical blocks (75.2 MB/71.7 MiB) [ 3.437687] vda: detected capacity change from 0 to 75194368 [ 3.456876] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.461220] vdb: detected capacity change from 0 to 1073741824 [ 3.482304] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.485715] vdc: detected capacity change from 0 to 2621440000 [ 3.500261] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.504077] vdd: detected capacity change from 0 to 2621440000 [ 3.522178] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.525486] vde: detected capacity change from 0 to 4294967296 [ 3.546902] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.550411] vdf: detected capacity change from 0 to 4294967296 [ 3.562347] libphy: Fixed MDIO Bus: probed [ 3.568274] usbcore: registered new interface driver usbserial_generic [ 3.570509] usbserial: USB Serial support registered for generic [ 3.572689] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.579129] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.582095] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.584699] mousedev: PS/2 mouse device common for all mice [ 3.588415] rtc_cmos 00:05: RTC can wake from S4 [ 3.591463] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.591571] rtc_cmos 00:05: registered as rtc0 [ 3.596200] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.599622] intel_pstate: CPU model not supported [ 3.602235] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.604563] hid: raw HID events driver (C) Jiri Kosina [ 3.608748] usbcore: registered new interface driver usbhid [ 3.610567] usbhid: USB HID core driver [ 3.612570] drop_monitor: Initializing network drop monitor service [ 3.614515] Initializing XFRM netlink socket [ 3.614707] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.616197] NET: Registered protocol family 10 [ 3.620619] Segment Routing with IPv6 [ 3.621733] NET: Registered protocol family 17 [ 3.623216] mpls_gso: MPLS GSO support [ 3.629113] RAS: Correctable Errors collector initialized. [ 3.630879] AVX version of gcm_enc/dec engaged. [ 3.631800] AES CTR mode by8 optimization enabled [ 3.700818] sched_clock: Marking stable (3700800619, 0)->(4737831576, -1037030957) [ 3.705393] registered taskstats version 1 [ 3.709770] Loading compiled-in X.509 certificates [ 3.713842] zswap: loaded using pool lzo/zbud [ 3.741053] Key type big_key registered [ 3.753344] Key type encrypted registered [ 3.755517] ima: No TPM chip found, activating TPM-bypass! [ 3.759085] ima: Allocated hash algorithm: sha1 [ 3.761887] ima: No architecture policies found [ 3.765517] evm: Initialising EVM extended attributes: [ 3.769206] evm: security.selinux [ 3.771513] evm: security.ima [ 3.773614] evm: security.capability [ 3.776103] evm: HMAC attrs: 0x1 [ 3.780174] rtc_cmos 00:05: setting system clock to 2026-08-27 17:41:46 UTC (1787852506) [ 3.786210] debug: unmapping init [mem 0xffffffffac203000-0xffffffffac3fffff] [ 3.789349] debug: unmapping init [mem 0xffffffffaaf82000-0xffffffffab258fff] [ 3.798063] Write protecting the kernel read-only data: 28672k [ 3.802754] debug: unmapping init [mem 0xffffffffa9603000-0xffffffffa97fffff] [ 3.805451] debug: unmapping init [mem 0xffffffffa9f14000-0xffffffffa9ffffff] [ 3.839177] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.847876] systemd[1]: Detected virtualization kvm. [ 3.849829] systemd[1]: Detected architecture x86-64. [ 3.851529] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.878712] systemd[1]: No hostname configured. [ 3.880808] systemd[1]: Set hostname to . [ 3.883442] random: systemd: uninitialized urandom read (16 bytes read) [ 3.886847] systemd[1]: Initializing machine ID from random generator. [ 3.933442] random: ln: uninitialized urandom read (6 bytes read) [ 4.041611] random: systemd: uninitialized urandom read (16 bytes read) [ 4.044810] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.049732] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.053496] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.704879] device-mapper: uevent: version 1.0.3 [ 4.707059] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.508660] virtio_net virtio0 ens2: renamed from eth0 [ 5.515468] random: fast init done [ 5.575491] scsi host0: ata_piix [ 5.606897] scsi host1: ata_piix [ 5.609088] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.612715] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.418542] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.284992] random: crng init done [ 10.287507] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.990769] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.286115] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.580821] SELinux: Disabled at runtime. [ 12.650694] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.660302] systemd[1]: Detected virtualization kvm. [ 12.662171] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.259639] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.263302] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.268201] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.272468] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.276486] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.285167] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.293939] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ 13.340910] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.803868] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.233561] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.295420] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.482152] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.501686] EDAC sbridge: Ver: 1.1.2 [ 16.508443] Key type dns_resolver registered [ 16.834387] NFS: Registering the id_resolver key type [ 16.836774] Key type id_resolver registered [ 16.838526] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg632-server login: [ 44.230633] libcfs: loading out-of-tree module taints kernel. [ 44.250553] Key type ._llcrypt registered [ 44.252556] Key type .llcrypt registered [ 44.323152] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_hostid [ 65.945246] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 69.043221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 69.060369] alg: No test for adler32 (adler32-zlib) [ 70.862631] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 72.393114] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 74.335669] Key type lgssc registered [ 76.285518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 94.236403] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 114.269209] hrtimer: interrupt took 1226965 ns [ 138.666483] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 153.501669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 153.522466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 154.713654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 154.739359] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 154.794943] Lustre: lustre-MDT0000: new disk, initializing [ 154.873631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 154.897773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 159.470681] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 174.013670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 174.197900] Lustre: 6534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 174.247965] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 174.251455] Lustre: Skipped 1 previous similar message [ 174.402746] Lustre: lustre-MDT0001: new disk, initializing [ 174.475090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 174.516117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 174.538876] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 179.622804] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.715306] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 196.764742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.015316] Lustre: lustre-OST0000: new disk, initializing [ 197.018676] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 197.023589] Lustre: 8471:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 197.075213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.773373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 202.791020] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 202.937257] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 205.027336] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.211285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 223.408759] Lustre: lustre-OST0001: new disk, initializing [ 223.411560] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 223.415995] Lustre: 9547:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 223.477085] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 230.169939] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.004957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 232.009789] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 232.057072] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 241.478191] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 246.242311] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 252.008144] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing check_logdir /tmp/testlogs/ [ 256.837864] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing yml_node [ 261.371118] Lustre: DEBUG MARKER: Client: 2.17.57.85 [ 263.906739] Lustre: DEBUG MARKER: MDS: 2.17.57.85 [ 266.389787] Lustre: DEBUG MARKER: OSS: 2.17.57.85 [ 268.140523] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 13:46:09 EDT 2026 [ 284.871466] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 287.326627] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 306.658061] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.660136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.669018] Lustre: Skipped 3 previous similar messages [ 306.675484] Lustre: Skipped 2 previous similar messages [ 309.683640] Lustre: server umount lustre-MDT0000 complete [ 313.264912] LustreError: 6523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852815 with bad export cookie 17786043615081786162 [ 313.265498] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 313.284587] LustreError: 6523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 316.897417] LustreError: 6541:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.898660] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.923037] LustreError: 6541:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 316.950910] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 316.955697] Lustre: Skipped 2 previous similar messages [ 318.948454] LustreError: 9297:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 318.948760] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 318.965745] LustreError: 9297:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 318.989411] Lustre: Skipped 1 previous similar message [ 319.988562] Lustre: server umount lustre-MDT0001 complete [ 324.658525] Lustre: server umount lustre-OST0000 complete [ 328.143517] Lustre: server umount lustre-OST0001 complete [ 334.650950] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_hostid [ 345.482984] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 379.141170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 379.319309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 379.345277] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 379.483724] Lustre: lustre-MDT0000: new disk, initializing [ 379.555264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 379.572670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 383.168294] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 394.592785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.669121] Lustre: 15100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 394.717313] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 394.722027] Lustre: Skipped 1 previous similar message [ 394.792387] Lustre: lustre-MDT0001: new disk, initializing [ 394.919944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 394.946234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 394.952573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 398.534354] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 408.020372] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 412.534417] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 417.708200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 417.982742] Lustre: lustre-OST0000: new disk, initializing [ 417.985994] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 417.995174] Lustre: 17067:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 418.082639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 419.305264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 419.327263] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 419.436746] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 423.717041] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 432.218851] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 436.776653] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 436.982204] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 441.390182] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 441.607163] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 444.903475] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 444.910456] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 444.918825] Lustre: Skipped 1 previous similar message [ 444.922679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 444.932126] Lustre: Skipped 1 previous similar message [ 449.679750] Lustre: server umount lustre-OST0000 complete [ 455.138390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.161681] Lustre: Skipped 1 previous similar message [ 455.165500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 459.709698] Lustre: server umount lustre-MDT0000 complete [ 460.256182] LustreError: 15111:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 460.267774] LustreError: 15111:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 462.851372] LustreError: 15093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852965 with bad export cookie 17786043615081787695 [ 462.855082] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 462.858521] LustreError: 15093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 463.109669] Lustre: server umount lustre-MDT0001 complete [ 466.705186] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 13:49:27 (1787852967) [ 469.001452] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 470.691175] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 13:49:31 (1787852971) [ 475.811938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 476.155272] LustreError: 19477:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 476.217908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 479.578161] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 481.251527] LustreError: 19478:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.318869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 493.197469] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 500.643373] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 505.886755] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 511.605519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 517.055034] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 523.848657] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 531.425885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.427493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 531.429409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 531.429417] Lustre: Skipped 1 previous similar message [ 531.444773] Lustre: Skipped 2 previous similar messages [ 531.448813] LustreError: Skipped 1 previous similar message [ 536.751512] Lustre: server umount lustre-OST0000 complete [ 541.665269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 546.790317] LustreError: 21273:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.801728] LustreError: 21273:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 546.868844] Lustre: server umount lustre-MDT0000 complete [ 550.963854] LustreError: 19457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853053 with bad export cookie 17786043615081788584 [ 550.967545] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 550.976553] LustreError: 19457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 551.762661] Lustre: server umount lustre-MDT0001 complete [ 564.581785] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 567.379041] Key type lgssc unregistered [ 567.652992] LNet: 23013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 567.663366] LNetError: 23013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 567.677276] LNet: Removed LNI 192.168.206.132@tcp [ 568.653188] Key type .llcrypt unregistered [ 568.655205] Key type ._llcrypt unregistered [ 590.707272] Key type ._llcrypt registered [ 590.709254] Key type .llcrypt registered [ 590.788635] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 591.273907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 591.341810] alg: No test for adler32 (adler32-zlib) [ 592.400496] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 592.628928] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 594.295184] Key type lgssc registered [ 595.278265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 602.563574] loop: module loaded [ 602.630865] loop0: detected capacity change from 0 to 204800000 [ 602.841413] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 602.860967] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 603.253057] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 608.542092] loop0: detected capacity change from 0 to 204800000 [ 608.689525] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 608.707941] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 608.723749] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 613.772723] loop0: detected capacity change from 0 to 204800000 [ 619.273637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 619.296716] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 620.606377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 620.672824] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 620.769842] Lustre: lustre-MDT0000: new disk, initializing [ 620.868641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 620.890535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 626.011980] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 635.995346] loop1: detected capacity change from 0 to 204800000 [ 641.628549] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 641.928346] Lustre: lustre-OST07c6: new disk, initializing [ 641.935938] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 641.943126] Lustre: Skipped 1 previous similar message [ 641.948183] Lustre: 26199:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 642.022082] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 648.015842] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 648.295523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 648.314261] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 648.358253] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 652.689315] Lustre: 25186:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 652.702236] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 652.708742] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 652.715476] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 652.721477] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 652.728055] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 667.492760] Lustre: 25186:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 667.502843] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 667.508381] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 667.518771] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 667.524051] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 667.528916] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 672.713901] Lustre: 25188:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 672.725973] Lustre: 25188:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 672.730222] Lustre: 25188:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 672.739841] Lustre: 25188:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 672.747012] Lustre: 25188:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 672.753240] Lustre: 25188:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 672.762875] Lustre: 25188:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 672.771413] Lustre: 25188:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 672.781128] Lustre: 25188:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 672.797135] Lustre: 25188:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 672.802984] Lustre: 25188:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 672.809127] Lustre: 25188:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 676.034624] Lustre: 26567:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 676.047697] Lustre: 26567:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 676.052680] Lustre: 26567:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 676.065874] Lustre: 26567:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 676.080049] Lustre: 26567:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 676.096060] Lustre: 26567:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 676.102590] Lustre: 26567:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 676.109519] Lustre: 26567:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 676.121707] Lustre: 26567:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 676.133144] Lustre: 26567:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 676.138495] Lustre: 26567:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 676.142757] Lustre: 26567:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 680.714683] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 680.723343] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 680.732591] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 680.744681] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 680.753810] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 680.765987] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 680.773147] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 680.782492] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 680.798652] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 680.815107] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 680.835424] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 680.843639] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 692.079382] Lustre: 25188:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 692.088634] Lustre: 25188:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 692.094507] Lustre: 25188:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 692.100482] Lustre: 25188:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 692.106514] Lustre: 25188:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 692.112786] Lustre: 25188:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 692.117747] Lustre: 25188:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 692.122543] Lustre: 25188:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 692.127894] Lustre: 25188:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 692.133277] Lustre: 25188:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 692.139646] Lustre: 25188:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 692.152603] Lustre: 25188:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 745.440896] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 745.455628] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 745.475464] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 747.730281] Lustre: server umount lustre-OST07c6 complete [ 757.362495] Lustre: server umount lustre-MDT0000 complete [ 771.531097] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 774.235502] Key type lgssc unregistered [ 774.499833] LNet: 28758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 774.518204] LNetError: 28758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 774.540745] LNet: Removed LNI 192.168.206.132@tcp [ 775.277668] Key type .llcrypt unregistered [ 775.282555] Key type ._llcrypt unregistered [ 793.255841] Key type ._llcrypt registered [ 793.258967] Key type .llcrypt registered [ 793.354355] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 794.372965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 794.394141] alg: No test for adler32 (adler32-zlib) [ 795.420791] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 795.633813] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 797.311216] Key type lgssc registered [ 798.906542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 806.921085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 806.940421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 808.647065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 814.299888] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 819.170420] LustreError: 30030:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 819.186383] LustreError: 30030:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 823.477085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 824.130591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 828.631368] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 836.112379] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 841.954578] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 848.786613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 848.866279] LustreError: 31851:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 849.106483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 854.277606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 855.138361] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 863.201167] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 880.106900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 880.111384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.114499] LustreError: Skipped 1 previous similar message [ 880.116507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 880.138132] Lustre: Skipped 1 previous similar message [ 881.922657] Lustre: server umount lustre-OST0000 complete [ 885.736217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 885.741301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 885.752633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 885.758287] Lustre: Skipped 1 previous similar message [ 890.342147] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 890.343289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 890.389944] Lustre: Skipped 1 previous similar message [ 891.881672] Lustre: server umount lustre-MDT0000 complete [ 895.460185] LustreError: 30031:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 895.499683] LustreError: 30031:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 895.610157] LustreError: 30011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853398 with bad export cookie 1499041545548438876 [ 895.612870] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 895.627315] LustreError: 30011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 895.961948] Lustre: server umount lustre-MDT0001 complete [ 906.663103] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 909.149878] Key type lgssc unregistered [ 909.487800] LNet: 33769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.507881] LNetError: 33769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.531486] LNet: Removed LNI 192.168.206.132@tcp [ 910.386681] Key type .llcrypt unregistered [ 910.395510] Key type ._llcrypt unregistered [ 928.808283] Key type ._llcrypt registered [ 928.810570] Key type .llcrypt registered [ 928.883805] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 930.005856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 930.038976] alg: No test for adler32 (adler32-zlib) [ 931.152850] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 931.359588] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 933.055233] Key type lgssc registered [ 934.437237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 941.439412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 941.458552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 942.817666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 947.009575] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 952.290402] LustreError: 35040:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.299603] LustreError: 35040:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 954.008238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 954.355446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 958.432600] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 965.839700] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 971.319392] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 978.096970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 978.511370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 985.604216] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 987.844268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 993.180637] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1003.499279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1003.505196] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1003.526030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1003.527152] Lustre: Skipped 1 previous similar message [ 1008.611458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1008.622924] Lustre: Skipped 2 previous similar messages [ 1008.963401] Lustre: server umount lustre-OST0000 complete [ 1013.729876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1013.738728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1013.742346] Lustre: Skipped 1 previous similar message [ 1018.851393] LustreError: 35040:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1018.904890] Lustre: server umount lustre-MDT0000 complete [ 1022.318861] LustreError: 35018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853525 with bad export cookie 3614673142638485980 [ 1022.327663] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1022.343105] LustreError: 35018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1022.657416] Lustre: server umount lustre-MDT0001 complete [ 1033.822157] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 1036.419740] Key type lgssc unregistered [ 1036.715409] LNet: 38576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.731453] LNetError: 38576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1036.759161] LNet: Removed LNI 192.168.206.132@tcp [ 1037.533449] Key type .llcrypt unregistered [ 1037.536222] Key type ._llcrypt unregistered [ 1060.075545] Key type ._llcrypt registered [ 1060.077792] Key type .llcrypt registered [ 1060.165901] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 1061.418792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1061.442194] alg: No test for adler32 (adler32-zlib) [ 1062.546099] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 1062.893383] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 1064.631683] Key type lgssc registered [ 1065.602354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1073.440594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1073.455266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1074.834683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1079.046792] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 1084.390584] LustreError: 39994:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1084.416618] LustreError: 39994:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1087.224697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1087.555762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1090.791455] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 1096.646429] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1102.055767] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1109.016475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1109.320397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1115.282771] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 1119.600045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1121.817791] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1128.417673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1128.425683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1128.444443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1129.806564] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.32@tcp (stopping) [ 1129.961270] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1129.999337] Lustre: Skipped 1 previous similar message [ 1130.159054] Lustre: server umount lustre-MDT0000 complete [ 1133.571231] LustreError: 39977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853636 with bad export cookie 57801716535535135 [ 1133.578373] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1133.582793] LustreError: 39977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1134.897083] LustreError: 39989:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1134.920420] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.32@tcp (stopping) [ 1134.931359] Lustre: Skipped 3 previous similar messages [ 1135.076061] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1138.084477] LustreError: 39989:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1138.084859] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1138.103513] LustreError: 39989:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1138.108578] Lustre: Skipped 1 previous similar message [ 1140.062465] Lustre: server umount lustre-MDT0001 complete [ 1169.376636] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1169.580550] Lustre: server umount lustre-OST0000 complete [ 1187.348555] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 1189.998100] Key type lgssc unregistered [ 1190.427577] LNet: 43871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1190.437419] LNetError: 43871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1190.449715] LNet: Removed LNI 192.168.206.132@tcp [ 1191.407348] Key type .llcrypt unregistered [ 1191.409226] Key type ._llcrypt unregistered [ 1212.588881] Key type ._llcrypt registered [ 1212.590868] Key type .llcrypt registered [ 1212.680216] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 1213.862109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1213.884117] alg: No test for adler32 (adler32-zlib) [ 1214.892221] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 1215.121567] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 1216.847151] Key type lgssc registered [ 1217.965801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1225.019578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1225.038754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1226.484504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1230.101735] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 1236.968685] LustreError: 45268:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1236.983031] LustreError: 45268:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1237.742893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1238.061332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1241.581445] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 1248.291406] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1253.724597] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1259.536738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1259.852914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1264.980836] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 1271.478941] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1272.180257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1282.386782] Lustre: lustre-OST0000: Not available for connect from 192.168.206.32@tcp (stopping) [ 1282.533551] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1282.537830] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1287.002838] Lustre: server umount lustre-OST0000 complete [ 1308.129235] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1308.138081] Lustre: Skipped 1 previous similar message [ 1308.144537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1308.147664] Lustre: Skipped 3 previous similar messages [ 1310.953676] Lustre: server umount lustre-MDT0000 complete [ 1313.249367] LustreError: 45262:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1313.269517] LustreError: 45262:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1314.228805] LustreError: 45248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853816 with bad export cookie 9409224193752355509 [ 1314.231594] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1314.244184] LustreError: 45248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1320.590113] Lustre: server umount lustre-MDT0001 complete [ 1331.692860] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 1334.200619] Key type lgssc unregistered [ 1334.509807] LNet: 48981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1334.515269] LNetError: 48981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1334.529675] LNet: Removed LNI 192.168.206.132@tcp [ 1335.297405] Key type .llcrypt unregistered [ 1335.299978] Key type ._llcrypt unregistered [ 1363.040245] Key type ._llcrypt registered [ 1363.048852] Key type .llcrypt registered [ 1363.253062] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 1364.250904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1364.357047] alg: No test for adler32 (adler32-zlib) [ 1365.458553] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 1365.754612] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 1367.512602] Key type lgssc registered [ 1368.775333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1376.827994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1376.866793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1378.345212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1382.676119] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 1388.521875] LustreError: 50399:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1388.545944] LustreError: 50399:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1393.632264] LustreError: 50400:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1393.733150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1394.162143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1400.308958] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 1409.785328] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1416.137841] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1422.254815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1422.670205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1423.755500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1429.114781] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 1435.899795] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1444.843803] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1448.445427] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1460.432467] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2714h06m00s [ 1462.015436] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1464.809458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1464.814066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1464.821688] Lustre: Skipped 2 previous similar messages [ 1464.838794] Lustre: Skipped 2 previous similar messages [ 1465.889291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1465.906506] Lustre: Skipped 2 previous similar messages [ 1468.753295] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.32@tcp (stopping) [ 1470.440187] Lustre: server umount lustre-MDT0000 complete [ 1471.972916] LustreError: 50394:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1471.988088] LustreError: 50394:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1473.740591] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1475.892157] LustreError: 50395:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1480.576792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1480.683118] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1480.906821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1480.973226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1481.964291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1485.440224] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 1489.190715] Lustre: DEBUG MARKER: Wait for df (36734) ... [ 1490.831170] Lustre: DEBUG MARKER: done [ 1495.521860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1495.536417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1495.548457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1495.554294] Lustre: Skipped 1 previous similar message [ 1496.545682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1496.555647] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1501.492840] Lustre: server umount lustre-OST0000 complete [ 1507.812394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1507.827056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1507.827347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1507.827351] Lustre: Skipped 3 previous similar messages [ 1507.827961] Lustre: Skipped 1 previous similar message [ 1511.876560] Lustre: server umount lustre-MDT0000 complete [ 1514.978210] LustreError: 50400:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1514.995173] LustreError: 50400:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 1515.719744] LustreError: 50381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854018 with bad export cookie 2467117513404587135 [ 1515.720729] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1515.737196] LustreError: 50381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1516.149394] Lustre: server umount lustre-MDT0001 complete [ 1529.006707] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 1531.857546] Key type lgssc unregistered [ 1532.140599] LNet: 55326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1532.150187] LNetError: 55326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1532.182374] LNet: Removed LNI 192.168.206.132@tcp [ 1533.109639] Key type .llcrypt unregistered [ 1533.111931] Key type ._llcrypt unregistered [ 1577.606093] Key type ._llcrypt registered [ 1577.615203] Key type .llcrypt registered [ 1577.765635] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 1579.030180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1579.061901] alg: No test for adler32 (adler32-zlib) [ 1580.074270] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 1580.463974] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 1582.167159] Key type lgssc registered [ 1583.394187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1593.195430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1593.231785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1594.658586] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1594.752519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1595.084518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1600.601753] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 1612.079274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1612.290560] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1612.340175] Lustre: 57533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1612.706279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1617.673086] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 1626.247742] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1631.602758] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1638.422888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1638.602906] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1638.777241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1644.694848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1644.965468] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 1653.570769] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1661.227586] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2714h09m21s [ 1662.658262] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1671.052594] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1673.995079] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1674.192441] LustreError: 57541:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1694.534527] Lustre: lustre-MDT0000: Client 3ccf3a8b-0d67-40b6-99a5-bf18b6c1fa3c (at 192.168.206.32@tcp) reconnecting [ 1714.215688] LustreError: 57541:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1716.557795] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1719.388684] Lustre: DEBUG MARKER: done [ 1726.432801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1726.444174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1726.454223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1726.944722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1726.950709] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1726.966484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1730.024163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1730.036024] Lustre: Skipped 1 previous similar message [ 1732.605042] Lustre: server umount lustre-OST0000 complete [ 1740.258891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1740.279791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1740.288079] Lustre: Skipped 1 previous similar message [ 1742.567836] Lustre: server umount lustre-MDT0000 complete [ 1746.198533] LustreError: 60297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854248 with bad export cookie 2328182659383281951 [ 1746.203274] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1746.215962] LustreError: 60297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1746.598434] Lustre: server umount lustre-MDT0001 complete [ 1757.516930] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 1760.589822] Key type lgssc unregistered [ 1760.961804] LNet: 61868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1760.974627] LNetError: 61868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1761.011349] LNet: Removed LNI 192.168.206.132@tcp [ 1761.897261] Key type .llcrypt unregistered [ 1761.900691] Key type ._llcrypt unregistered [ 1803.661500] Key type ._llcrypt registered [ 1803.673235] Key type .llcrypt registered [ 1803.792490] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 1805.048797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1805.067637] alg: No test for adler32 (adler32-zlib) [ 1806.316610] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 1806.598646] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 1808.303148] Key type lgssc registered [ 1809.542601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1813.059334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1813.101143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1814.416538] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1814.454393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1814.725601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1815.649602] Lustre: Failing over lustre-MDT0000 [ 1815.956541] Lustre: server umount lustre-MDT0000 complete [ 1823.012058] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 14:12:03 (1787854323) [ 1828.853601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1829.203030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1833.942686] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 1843.285118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1843.372102] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1843.398888] Lustre: 64664:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1843.782495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1848.713556] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 1857.523125] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1863.448405] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1869.967897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1870.257661] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1870.471815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1876.076250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1876.457193] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 1884.209993] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1891.437111] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1900.002517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1900.013206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1900.023631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1902.048955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1902.049962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1902.075065] Lustre: Skipped 1 previous similar message [ 1902.101574] Lustre: Skipped 2 previous similar messages [ 1904.721941] Lustre: server umount lustre-MDT0000 complete [ 1909.823556] LustreError: 65402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854412 with bad export cookie 4292085175089666754 [ 1909.832503] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1909.850590] LustreError: 65402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1910.292821] Lustre: server umount lustre-MDT0001 complete [ 1914.542911] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1918.835403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1926.422586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1928.607125] Lustre: 63588:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787854415/real 1787854415] req@00000000f3361eda x1874701119956096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787854431 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1928.645652] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1930.719939] Lustre: 63591:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787854417/real 1787854417] req@00000000516d3cf2 x1874701119956224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787854433 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1933.791374] Lustre: 63590:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787854420/real 1787854420] req@000000006846c72c x1874701119956352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787854436 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1934.817525] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3b908cb9eef0a3a5 [ 1934.838869] Lustre: MGC192.168.206.132@tcp: Connection restored to 0@lo (at 0@lo) [ 1935.064042] LustreError: 66520:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1935.077582] LustreError: 66520:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 1935.163557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1935.264058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1936.162536] Lustre: 63588:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787854422/real 1787854422] req@00000000d85c5c66 x1874701119956608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787854438 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1937.197274] LustreError: 66521:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1939.234541] LustreError: 68442:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1939.265675] LustreError: 68442:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1939.406183] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 1941.473485] Lustre: 63589:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787854427/real 1787854427] req@0000000024bcb4bc x1874701119956736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787854443 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1944.561822] LustreError: 68440:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1944.571019] LustreError: 68440:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1948.305574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1948.644666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1952.725623] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 1953.785534] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1957.863861] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1960.349880] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1966.744620] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1978.342584] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1978.354836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1978.373238] Lustre: Skipped 2 previous similar messages [ 1978.387148] Lustre: Skipped 2 previous similar messages [ 1981.871617] Lustre: server umount lustre-MDT0000 complete [ 1983.459161] LustreError: 68971:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1983.477661] LustreError: 68971:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1985.105806] LustreError: 65402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854487 with bad export cookie 4292085175089669029 [ 1985.109101] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1985.115389] LustreError: 65402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1985.401178] Lustre: server umount lustre-MDT0001 complete [ 1991.256581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2000.699279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2004.960505] Lustre: 63588:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787854491/real 1787854491] req@0000000019fddb71 x1874701119984512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787854507 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2004.980894] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2010.144982] LustreError: 63587:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009795c34e x1874701119985664/t0(0) o250->MGC192.168.206.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2010.599853] LustreError: 66521:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2010.670163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2010.722510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2015.277988] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 2023.638164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2028.543569] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 2029.555013] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2033.636556] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2036.591238] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2042.258205] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2048.994947] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2048.999602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2049.020591] Lustre: Skipped 1 previous similar message [ 2049.030421] Lustre: Skipped 2 previous similar messages [ 2054.117223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2054.125985] Lustre: Skipped 2 previous similar messages [ 2054.801488] Lustre: server umount lustre-MDT0000 complete [ 2058.439720] LustreError: 65402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854561 with bad export cookie 4292085175089670408 [ 2058.440855] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2058.453561] LustreError: 65402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2058.820783] Lustre: server umount lustre-MDT0001 complete [ 2064.903882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2068.073429] Lustre: DEBUG MARKER: files compared the same [ 2080.215880] Lustre: server umount lustre-OST0000 complete [ 2097.191642] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 2099.989667] Key type lgssc unregistered [ 2100.299260] LNet: 74404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2100.306547] LNetError: 74404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2100.336716] LNet: Removed LNI 192.168.206.132@tcp [ 2101.248172] Key type .llcrypt unregistered [ 2101.251797] Key type ._llcrypt unregistered [ 2125.764597] Key type ._llcrypt registered [ 2125.768803] Key type .llcrypt registered [ 2125.890412] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 2126.857371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2126.943198] alg: No test for adler32 (adler32-zlib) [ 2128.049106] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 2128.263864] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 2129.919154] Key type lgssc registered [ 2131.165844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2138.806960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2138.851970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2140.330125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2144.843786] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug +malloc all [ 2149.856297] LustreError: 75823:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2149.873427] LustreError: 75823:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2153.506582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2153.943673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2158.902634] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug +malloc all [ 2166.239879] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2172.246363] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2179.590413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2180.038449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2185.262636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2187.320917] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug +malloc all [ 2195.143813] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2208.736422] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2208.747676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2208.760877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2208.788027] Lustre: Skipped 1 previous similar message [ 2208.825425] Lustre: Skipped 1 previous similar message [ 2213.891734] Lustre: server umount lustre-OST0000 complete [ 2218.977163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2218.999259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2219.004455] Lustre: Skipped 1 previous similar message [ 2223.648600] Lustre: server umount lustre-MDT0000 complete [ 2224.100873] LustreError: 75824:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2224.118780] LustreError: 75824:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2227.374361] LustreError: 75804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854730 with bad export cookie 16844151949390546745 [ 2227.377520] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2227.387467] LustreError: 75804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2227.660899] Lustre: server umount lustre-MDT0001 complete [ 2239.103530] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 2241.583502] Key type lgssc unregistered [ 2241.883250] LNet: 79371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2241.889853] LNetError: 79371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2241.912353] LNet: Removed LNI 192.168.206.132@tcp [ 2242.783278] Key type .llcrypt unregistered [ 2242.787198] Key type ._llcrypt unregistered [ 2268.368400] Key type ._llcrypt registered [ 2268.371195] Key type .llcrypt registered [ 2268.519119] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 2269.522961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2269.557113] alg: No test for adler32 (adler32-zlib) [ 2270.612875] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 2270.791149] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 2272.432197] Key type lgssc registered [ 2273.544653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2280.617842] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2280.649845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2296.223817] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2301.343596] LustreError: 80762:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.132@tcp: failed processing log, type 4: rc = -110 [ 2328.031437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2333.498458] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 2340.779795] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2347.897778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2348.048738] Lustre: *** cfs_fail_loc=706, val=0*** [ 2348.063486] LustreError: 80098:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000aa9aa346 x1874701606455552/t0(0) o250->MGC192.168.206.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2348.474305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2348.560809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2352.579826] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 2353.635336] LustreError: 80787:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2353.647320] LustreError: 80787:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2358.757045] LustreError: 80788:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2359.930909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2360.293499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2364.172468] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 2371.030690] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2376.357559] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2379.234570] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2379.244308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2379.259951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2380.771497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2380.777920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2380.799841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2382.826397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2382.838873] Lustre: Skipped 1 previous similar message [ 2384.440123] Lustre: server umount lustre-OST0000 complete [ 2387.936818] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2387.944204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2387.947404] Lustre: Skipped 1 previous similar message [ 2393.056644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2393.061758] Lustre: Skipped 2 previous similar messages [ 2402.271126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2402.415546] Lustre: server umount lustre-MDT0000 complete [ 2403.296903] LustreError: 81851:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.309318] LustreError: 81851:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2405.585243] LustreError: 80769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854908 with bad export cookie 3308045821565280722 [ 2405.589238] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2405.600182] LustreError: 80769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2405.848530] Lustre: server umount lustre-MDT0001 complete [ 2417.164494] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 2420.935946] Key type lgssc unregistered [ 2421.445682] LNet: 84381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2421.469079] LNetError: 84381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2421.488575] LNet: Removed LNI 192.168.206.132@tcp [ 2422.223374] Key type .llcrypt unregistered [ 2422.225038] Key type ._llcrypt unregistered [ 2449.037449] Key type ._llcrypt registered [ 2449.039417] Key type .llcrypt registered [ 2449.136751] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 2449.904907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2449.942620] alg: No test for adler32 (adler32-zlib) [ 2450.884317] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 2451.038622] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 2452.687167] Key type lgssc registered [ 2453.588985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2460.435550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2460.477761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2465.217567] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 2472.070437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2472.539929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2476.515631] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 2477.543814] LustreError: 86488:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2477.565361] LustreError: 86488:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2482.667127] LustreError: 86492:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.686577] LustreError: 86492:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2484.154261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2484.423025] LustreError: 86488:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2484.442291] LustreError: 86488:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2484.469318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2489.829841] LustreError: 86486:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.857397] LustreError: 86486:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2490.389131] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 2494.952855] LustreError: 87234:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2494.976454] LustreError: 87234:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2498.146798] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2498.160076] Lustre: MGS: Client 6354a75b-011d-4fa0-acf0-4d0269a1f1f1 (at 0@lo) reconnecting [ 2498.184294] LustreError: 86502:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2498.442316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2498.513640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2502.558605] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 2519.012581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2519.015765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2519.018747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2519.027724] Lustre: Skipped 1 previous similar message [ 2521.183934] Lustre: server umount lustre-OST0000 complete [ 2529.250339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2529.250875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2529.259807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2529.269146] Lustre: Skipped 1 previous similar message [ 2529.287971] Lustre: Skipped 2 previous similar messages [ 2530.530216] Lustre: server umount lustre-MDT0000 complete [ 2534.369962] LustreError: 86487:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.391835] LustreError: 86487:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 2534.709950] Lustre: server umount lustre-MDT0001 complete [ 2537.841839] Lustre: server umount MGS complete [ 2547.814884] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 14:24:08 (1787855048) [ 2577.422115] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_hostid [ 2590.028767] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 2625.922109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.364684] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 2641.423329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2641.614832] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2641.654846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2641.724196] Lustre: lustre-MDT0001: new disk, initializing [ 2641.803991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2641.833105] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2641.842468] Lustre: 93656:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2641.860531] LustreError: 93656:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2645.290817] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 2654.916288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2655.100963] Lustre: lustre-OST0000: new disk, initializing [ 2655.104328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2655.109095] Lustre: 94422:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2655.169586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2661.191125] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 2661.409613] Lustre: 94717:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000059acc54 x1874701795236736/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787855180 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2670.194323] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2676.636136] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2676.648951] Lustre: MGS: Client db542c17-06ae-4d08-bdb5-ee114985a44b (at 0@lo) reconnecting [ 2676.701936] Lustre: 95467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2676.737281] LustreError: 93642:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2676.748608] LustreError: 93642:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2676.760197] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2676.867892] Lustre: lustre-MDT0000: new disk, initializing [ 2677.028233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2677.053522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2677.066676] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2678.112385] Lustre: 95491:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000f9f1ae7c x1874701795244288/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787855196 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2681.768252] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 2683.233894] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2687.211574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2687.219911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2687.312170] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2713.056101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2713.066562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2713.086062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2713.089496] Lustre: Skipped 1 previous similar message [ 2713.104573] Lustre: Skipped 1 previous similar message [ 2718.177415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2718.192030] Lustre: Skipped 1 previous similar message [ 2723.309774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2723.318804] Lustre: Skipped 1 previous similar message [ 2726.367757] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2726.542612] Lustre: server umount lustre-OST0000 complete [ 2733.540451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2733.545164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2733.547460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2733.566827] Lustre: Skipped 1 previous similar message [ 2744.287541] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2744.417882] Lustre: server umount lustre-MDT0000 complete [ 2763.231154] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2763.434408] Lustre: server umount lustre-MDT0001 complete [ 2767.077985] Lustre: server umount MGS complete [ 2776.793873] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 14:27:57 (1787855277) [ 2792.564510] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 2795.220906] Key type lgssc unregistered [ 2795.464533] LNet: 98008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2795.470784] LNetError: 98008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2795.489149] LNet: Removed LNI 192.168.206.132@tcp [ 2796.153818] Key type .llcrypt unregistered [ 2796.163927] Key type ._llcrypt unregistered [ 2812.491362] Key type ._llcrypt registered [ 2812.493208] Key type .llcrypt registered [ 2812.594672] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 2813.389785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2813.450516] alg: No test for adler32 (adler32-zlib) [ 2814.698536] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 2814.947584] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 2816.655169] Key type lgssc registered [ 2817.769520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2825.306112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2825.329757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2826.856039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2830.738572] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 2836.472735] LustreError: 99231:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2836.493511] LustreError: 99231:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2838.251244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2838.609864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2843.008489] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 2850.812898] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2858.172941] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2865.495117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2865.961160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2871.832328] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 2872.108199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2879.889514] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2892.776421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2892.786468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2892.799100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2893.798498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2893.807371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2893.808111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2896.822700] Lustre: server umount lustre-OST0000 complete [ 2904.033391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2904.037391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2904.044721] Lustre: Skipped 1 previous similar message [ 2904.058721] Lustre: Skipped 2 previous similar messages [ 2906.337487] Lustre: server umount lustre-MDT0000 complete [ 2909.164535] LustreError: 99226:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2909.192562] LustreError: 99226:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2909.323748] LustreError: 99211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855412 with bad export cookie 13667039223683900290 [ 2909.335718] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2909.568615] Lustre: server umount lustre-MDT0001 complete [ 2920.135510] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 2923.734344] Key type lgssc unregistered [ 2924.204595] LNet: 102766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2924.218093] LNetError: 102766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2924.236208] LNet: Removed LNI 192.168.206.132@tcp [ 2925.179501] Key type .llcrypt unregistered [ 2925.182940] Key type ._llcrypt unregistered [ 2944.153581] Key type ._llcrypt registered [ 2944.155522] Key type .llcrypt registered [ 2944.255568] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 2945.023620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2945.169171] alg: No test for adler32 (adler32-zlib) [ 2946.423982] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 2946.776760] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 2948.535495] Key type lgssc registered [ 2949.964611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2955.788156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2955.794700] LustreError: 103849:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2955.807525] LustreError: 103850:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2955.819091] LustreError: 103849:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4991 [ 2955.827471] LustreError: 103849:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2955.834572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2955.844227] LustreError: 103849:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2957.259657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2964.096400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2964.329241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2968.273667] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 2972.785447] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing lsmod [ 2975.243764] LustreError: 105049:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2975.261774] LustreError: 105056:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2975.266458] LustreError: 105049:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4988 [ 2975.273338] LustreError: 105049:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2975.282483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2975.285739] LustreError: 105049:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2975.728050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2978.283328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2978.285575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2978.302301] Lustre: Skipped 2 previous similar messages [ 2978.307948] Lustre: Skipped 1 previous similar message [ 2983.392428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2983.397804] Lustre: Skipped 3 previous similar messages [ 2984.647923] Lustre: server umount lustre-MDT0000 complete [ 2987.821063] LustreError: 103860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855490 with bad export cookie 13502979506721587613 [ 2987.823198] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2987.833290] LustreError: 103860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2987.852640] LustreError: 103885:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2987.854089] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2987.855530] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2987.880232] LustreError: 103885:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2988.050588] Lustre: server umount lustre-MDT0001 complete [ 2992.111479] Lustre: server umount lustre-OST0000 complete [ 3000.953920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3001.225587] LustreError: 106118:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3001.275918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3005.932369] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 3006.441140] LustreError: 106119:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3010.530050] LustreError: 106118:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3013.383944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3013.834809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3018.052461] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 3025.594087] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3031.473920] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3038.322655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3038.694894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3044.239624] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 3045.876890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3051.483653] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3059.172766] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3059.187432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3059.218194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3061.218378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3061.235852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3065.383522] Lustre: server umount lustre-OST0000 complete [ 3070.434248] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3070.444581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3070.463550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3070.474467] Lustre: Skipped 2 previous similar messages [ 3081.699604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3081.709421] Lustre: Skipped 5 previous similar messages [ 3084.255121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3084.558985] Lustre: server umount lustre-MDT0000 complete [ 3086.819595] LustreError: 107353:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3086.831254] LustreError: 107353:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3088.625837] LustreError: 106100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855591 with bad export cookie 13502979506721588292 [ 3088.631088] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3088.635872] LustreError: 106100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3089.088940] Lustre: server umount lustre-MDT0001 complete [ 3099.886909] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 3102.373694] Key type lgssc unregistered [ 3102.697578] LNet: 109655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3102.711258] LNetError: 109655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3102.727849] LNet: Removed LNI 192.168.206.132@tcp [ 3103.593181] Key type .llcrypt unregistered [ 3103.596622] Key type ._llcrypt unregistered [ 3127.049959] Key type ._llcrypt registered [ 3127.052221] Key type .llcrypt registered [ 3127.124194] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 3128.158890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3128.184307] alg: No test for adler32 (adler32-zlib) [ 3129.135343] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 3129.279470] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 3130.903202] Key type lgssc registered [ 3132.214207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3139.768898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3139.792319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3141.285995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3146.278343] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 3150.821227] LustreError: 111074:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3150.841155] LustreError: 111074:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3155.937123] LustreError: 111075:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3156.078116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3156.520802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3161.198740] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 3169.017153] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3175.288437] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3182.347804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3182.701705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3185.769828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3188.739496] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 3196.440745] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3206.336276] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3211.023993] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3216.864361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3216.879529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3216.912041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3217.891716] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3217.897165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3217.897899] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3217.975132] Lustre: Skipped 1 previous similar message [ 3221.508447] Lustre: server umount lustre-OST0000 complete [ 3228.131965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3228.135365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3228.146650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3228.179575] Lustre: Skipped 1 previous similar message [ 3233.252753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3233.256112] Lustre: Skipped 2 previous similar messages [ 3237.348778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3237.359258] Lustre: Skipped 1 previous similar message [ 3239.391345] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3239.558462] Lustre: server umount lustre-MDT0000 complete [ 3242.464216] LustreError: 112079:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3242.481262] LustreError: 112079:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3243.077748] LustreError: 111054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855745 with bad export cookie 9263479685444256498 [ 3243.079502] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3243.085176] LustreError: 111054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3243.448505] Lustre: server umount lustre-MDT0001 complete [ 3255.446759] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 3258.291211] Key type lgssc unregistered [ 3258.581772] LNet: 114719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3258.609402] LNetError: 114719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3258.627682] LNet: Removed LNI 192.168.206.132@tcp [ 3259.487848] Key type .llcrypt unregistered [ 3259.492415] Key type ._llcrypt unregistered [ 3278.640751] Key type ._llcrypt registered [ 3278.647478] Key type .llcrypt registered [ 3278.753259] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 3279.980286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3280.022356] alg: No test for adler32 (adler32-zlib) [ 3281.041179] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 3281.207990] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 3282.863259] Key type lgssc registered [ 3283.775963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3290.730070] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3290.766288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3292.327701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3297.079616] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 3301.866893] LustreError: 115943:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3301.895294] LustreError: 115943:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3306.977722] LustreError: 115942:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3307.318731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3307.646338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3312.441630] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 3321.346653] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3327.968784] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3336.140433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3336.573583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3337.604439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3343.624645] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 3352.898927] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3363.296632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3363.304651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3363.314682] LustreError: Skipped 1 previous similar message [ 3363.318368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3363.337157] Lustre: Skipped 1 previous similar message [ 3363.365388] Lustre: Skipped 1 previous similar message [ 3367.708653] Lustre: server umount lustre-OST0000 complete [ 3373.537228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3373.540750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3373.560398] Lustre: Skipped 1 previous similar message [ 3373.569450] Lustre: Skipped 1 previous similar message [ 3378.660500] LustreError: 118067:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3378.676830] LustreError: 118067:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3378.830110] Lustre: server umount lustre-MDT0000 complete [ 3382.672798] LustreError: 115924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855885 with bad export cookie 8730098528889710712 [ 3382.693881] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3383.138378] Lustre: server umount lustre-MDT0001 complete [ 3395.082970] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 3398.190543] Key type lgssc unregistered [ 3398.508707] LNet: 119479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3398.524432] LNetError: 119479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3399.589989] LNet: Removed LNI 192.168.206.132@tcp [ 3400.461751] Key type .llcrypt unregistered [ 3400.463361] Key type ._llcrypt unregistered [ 3423.935136] Key type ._llcrypt registered [ 3423.937727] Key type .llcrypt registered [ 3424.076133] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 3425.243761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3425.273262] alg: No test for adler32 (adler32-zlib) [ 3426.368931] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 3426.581837] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 3428.231245] Key type lgssc registered [ 3429.276690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3436.717255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3436.734112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3438.172449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3442.501941] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 3448.311203] LustreError: 120896:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3448.334710] LustreError: 120896:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3451.233934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3451.676270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3455.852532] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 3462.374796] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3468.183746] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3475.589587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3475.986058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3482.721420] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 3484.287808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3490.806938] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3496.928841] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3500.775388] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3500.786416] Lustre: Skipped 2 previous similar messages [ 3505.483459] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3505.493072] Lustre: Skipped 2 previous similar messages [ 3505.813351] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3505.840762] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3510.737310] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.32@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3510.745799] Lustre: Skipped 2 previous similar messages [ 3515.479484] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.32@tcp 192.168.0.[2,10]@tcp [ 3520.562637] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.32/24@tcp in log lustre-MDT0000 [ 3520.571449] Lustre: Skipped 2 previous similar messages [ 3522.026226] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.32/24@tcp [ 3522.034557] Lustre: Skipped 1 previous similar message [ 3530.218493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3530.218621] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3530.234565] LustreError: Skipped 2 previous similar messages [ 3530.266534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3530.739163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3530.766867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3531.251424] Lustre: server umount lustre-OST0000 complete [ 3535.845689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3535.864283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3535.870660] Lustre: Skipped 1 previous similar message [ 3540.964819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3540.991643] Lustre: Skipped 1 previous similar message [ 3541.521315] Lustre: server umount lustre-MDT0000 complete [ 3545.569196] LustreError: 123524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856048 with bad export cookie 1508128671445161730 [ 3545.572643] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3545.589237] LustreError: 123524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3546.430526] Lustre: server umount lustre-MDT0001 complete [ 3560.381839] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 3563.765668] Key type lgssc unregistered [ 3564.133984] LNet: 125590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3564.144442] LNetError: 125590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3564.169041] LNet: Removed LNI 192.168.206.132@tcp [ 3565.014837] Key type .llcrypt unregistered [ 3565.022467] Key type ._llcrypt unregistered [ 3597.819771] Key type ._llcrypt registered [ 3597.821285] Key type .llcrypt registered [ 3598.017775] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 3599.731172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3599.776781] alg: No test for adler32 (adler32-zlib) [ 3600.852767] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 3601.238423] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 3602.943152] Key type lgssc registered [ 3604.428851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3613.194691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3613.209480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3614.600828] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3614.636608] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.32/24@tcp [ 3614.663462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3620.113812] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 3624.931790] LustreError: 127168:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3624.949220] LustreError: 127168:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3630.057169] LustreError: 127169:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3630.363760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3630.647553] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3630.694280] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.32/24@tcp [ 3630.794581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3636.910702] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 3646.879104] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3653.760111] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3662.046621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3662.446188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3670.311677] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 3679.580992] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3686.080293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3701.728242] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3701.729716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3701.739111] LustreError: Skipped 1 previous similar message [ 3701.743078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3701.766262] Lustre: Skipped 1 previous similar message [ 3701.792690] Lustre: Skipped 1 previous similar message [ 3706.623980] Lustre: server umount lustre-OST0000 complete [ 3711.968929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3711.974259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3711.997672] Lustre: Skipped 1 previous similar message [ 3712.024218] Lustre: Skipped 1 previous similar message [ 3717.089515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3717.099480] Lustre: Skipped 1 previous similar message [ 3717.670500] Lustre: server umount lustre-MDT0000 complete [ 3722.159380] LustreError: 127148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856224 with bad export cookie 10080532806822618270 [ 3722.161824] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3722.175648] LustreError: 127148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3722.519805] Lustre: server umount lustre-MDT0001 complete [ 3735.835179] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 3739.198593] Key type lgssc unregistered [ 3739.551616] LNet: 130786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3739.561121] LNetError: 130786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3739.584699] LNet: Removed LNI 192.168.206.132@tcp [ 3740.588671] Key type .llcrypt unregistered [ 3740.593875] Key type ._llcrypt unregistered [ 3769.281951] Key type ._llcrypt registered [ 3769.283983] Key type .llcrypt registered [ 3769.418437] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_hostid [ 3783.879781] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 3784.742840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3784.785141] alg: No test for adler32 (adler32-zlib) [ 3785.854753] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 3786.096028] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 3787.799203] Key type lgssc registered [ 3789.185667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3828.244331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3828.276267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3829.510337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3829.541984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3829.618460] Lustre: lustre-MDT0000: new disk, initializing [ 3829.692531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3829.716976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3834.890703] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 3847.513447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3847.649436] Lustre: 134187:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3847.697854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3847.705617] Lustre: Skipped 1 previous similar message [ 3847.784684] Lustre: lustre-MDT0001: new disk, initializing [ 3847.869611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3847.909198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3847.917236] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3853.190295] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 3863.257303] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3868.924870] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3876.612478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3877.053207] Lustre: lustre-OST0000: new disk, initializing [ 3877.061505] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3877.076808] Lustre: 136156:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3877.216636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3884.003666] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 3885.177925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3885.197433] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3885.350978] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3894.751569] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3899.903321] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3900.129755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3905.836620] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3906.008604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3911.150456] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3911.155091] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3911.159270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3911.179394] Lustre: Skipped 1 previous similar message [ 3911.192215] LustreError: Skipped 1 previous similar message [ 3914.465553] Lustre: server umount lustre-OST0000 complete [ 3919.841565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3919.850949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3919.866525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3919.871866] Lustre: Skipped 1 previous similar message [ 3921.377848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3921.378675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3921.405338] Lustre: Skipped 1 previous similar message [ 3924.719962] Lustre: server umount lustre-MDT0000 complete [ 3928.495451] LustreError: 136155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856431 with bad export cookie 18285132891803625667 [ 3928.497267] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3928.516586] LustreError: 136155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3928.837872] Lustre: server umount lustre-MDT0001 complete [ 3937.888801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3938.318352] LustreError: 138372:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3938.334380] LustreError: 138372:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 3938.411034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3943.113491] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 3943.904773] LustreError: 138373:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3949.026931] LustreError: 138372:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3951.755218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3952.138935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3956.618670] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 3965.288092] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3972.105158] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3978.949598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3979.233726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3985.275478] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 3993.795841] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3999.388520] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3999.695695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4004.826047] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4005.155194] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4015.209626] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4015.355274] Lustre: lustre-OST0001: new disk, initializing [ 4015.358316] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4015.362820] Lustre: 141748:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4015.430044] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4021.868282] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 4025.513243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4025.529561] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4025.678385] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4029.650494] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4029.865283] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4034.229262] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4034.366469] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4039.416722] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff892844bbf800.ost_server_uuid 50 [ 4041.064568] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff892844bbf800.ost_server_uuid in IDLE state after 0 sec [ 4051.428728] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4051.434522] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4051.461791] LustreError: Skipped 1 previous similar message [ 4051.462560] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4051.484384] Lustre: Skipped 1 previous similar message [ 4051.513249] Lustre: Skipped 1 previous similar message [ 4051.762427] Lustre: server umount lustre-OST0001 complete [ 4056.547485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4056.554913] LustreError: 140199:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4056.560535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4056.588234] LustreError: 140199:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4061.667809] LustreError: 140197:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4061.683167] LustreError: 140197:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4061.824732] Lustre: server umount lustre-OST0000 complete [ 4066.793912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4066.799082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4066.805285] Lustre: Skipped 2 previous similar messages [ 4066.812343] Lustre: Skipped 1 previous similar message [ 4072.004446] Lustre: server umount lustre-MDT0000 complete [ 4076.346049] LustreError: 138354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856579 with bad export cookie 18285132891803626549 [ 4076.353079] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4076.359386] LustreError: 138354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4076.682773] Lustre: server umount lustre-MDT0001 complete [ 4096.849577] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 4099.668373] Key type lgssc unregistered [ 4099.913916] LNet: 144752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4099.929381] LNetError: 144752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4099.955250] LNet: Removed LNI 192.168.206.132@tcp [ 4100.798238] Key type .llcrypt unregistered [ 4100.800611] Key type ._llcrypt unregistered [ 4139.689129] Key type ._llcrypt registered [ 4139.692225] Key type .llcrypt registered [ 4139.781797] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_hostid [ 4155.025317] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 4156.412945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4156.439549] alg: No test for adler32 (adler32-zlib) [ 4157.434420] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 4157.739945] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 4159.471347] Key type lgssc registered [ 4160.674539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4196.055905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4196.090527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4197.334434] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4197.392825] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4197.445942] Lustre: lustre-MDT0000: new disk, initializing [ 4197.618094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4197.643535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4202.687652] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 4214.619722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4214.756443] Lustre: 148908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4214.829934] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4214.834944] Lustre: Skipped 1 previous similar message [ 4214.935942] Lustre: lustre-MDT0001: new disk, initializing [ 4215.050196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4215.094704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4215.107947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4219.312098] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 4228.267569] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4233.747856] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4240.159461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4240.443760] Lustre: lustre-OST0000: new disk, initializing [ 4240.447394] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4240.454529] Lustre: 150878:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4240.527566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4246.188875] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 4247.116814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4247.128157] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4247.259671] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4254.823982] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4260.139836] Lustre: Failing over lustre-OST0000 [ 4260.287626] Lustre: server umount lustre-OST0000 complete [ 4278.239316] Lustre: 147134:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787856764/real 1787856764] req@0000000079ced4cf x1874703585149440/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787856780 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4278.264087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4279.034133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4279.327990] Lustre: 147134:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787856765/real 1787856765] req@0000000007e2bee4 x1874703585150464/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787856781 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4279.330293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4279.349161] Lustre: 147134:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4279.411955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4279.429012] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4283.431541] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4284.447887] Lustre: 147134:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787856770/real 1787856770] req@0000000012529962 x1874703585151488/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787856786 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4284.478396] Lustre: 147134:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4285.423855] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 4286.561529] Lustre: 147134:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787856773/real 1787856773] req@000000007b3f55d1 x1874703585152384/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787856789 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4289.169389] Lustre: Failing over lustre-MDT0000 [ 4289.182031] LustreError: 147132:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4289.354771] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.32@tcp (stopping) [ 4289.397193] Lustre: lustre-OST0000: Recovery over after 0:06, of 3 clients 3 recovered and 0 were evicted. [ 4289.403406] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4289.584686] Lustre: server umount lustre-MDT0000 complete [ 4292.064233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4292.068547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4292.247166] Lustre: 147133:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787856778/real 1787856778] req@000000001f339385 x1874703585153408/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787856794 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4292.275873] Lustre: 147133:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4294.114108] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4294.130499] Lustre: Skipped 1 previous similar message [ 4299.233642] LustreError: 148920:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4299.256332] LustreError: 148920:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 8 previous similar messages [ 4304.364187] LustreError: 151598:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4304.375238] LustreError: 151598:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 4307.746552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4307.863526] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4308.030587] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4308.083668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4308.135152] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4309.808987] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4311.825576] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 4313.582178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4313.607641] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4313.681195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4319.205751] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4319.211993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4319.228967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4320.228789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4320.236233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4321.095756] Lustre: server umount lustre-OST0000 complete [ 4328.930397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4328.931762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4328.933145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4328.935876] Lustre: Skipped 2 previous similar messages [ 4330.645224] Lustre: server umount lustre-MDT0000 complete [ 4333.562628] LustreError: 148902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856836 with bad export cookie 10111812567978632076 [ 4333.564473] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4333.571053] LustreError: 148902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4333.926974] Lustre: server umount lustre-MDT0001 complete [ 4344.069077] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 4346.284927] Key type lgssc unregistered [ 4346.504891] LNet: 154535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4346.510920] LNetError: 154535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4346.535156] LNet: Removed LNI 192.168.206.132@tcp [ 4347.379796] Key type .llcrypt unregistered [ 4347.381980] Key type ._llcrypt unregistered [ 4382.969703] Key type ._llcrypt registered [ 4382.972786] Key type .llcrypt registered [ 4383.106996] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 4384.015782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4384.035123] alg: No test for adler32 (adler32-zlib) [ 4385.026222] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 4385.246556] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 4386.887169] Key type lgssc registered [ 4388.120333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4394.968790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4395.001882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4396.454787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4401.407821] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 4406.757255] LustreError: 156054:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4406.770555] LustreError: 156054:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4410.854465] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4411.874289] LustreError: 156055:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4414.565488] LustreError: 156083:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4414.575379] LustreError: 156083:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4414.582691] LustreError: 156083:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4414.889291] Lustre: server umount lustre-MDT0000 complete [ 4429.863494] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_hostid [ 4439.547969] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 4472.723532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4472.906333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4472.937137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4473.033842] Lustre: lustre-MDT0000: new disk, initializing [ 4473.119858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4473.133877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4476.857653] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 4486.902608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4487.058666] Lustre: 159387:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4487.132630] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4487.136765] Lustre: Skipped 1 previous similar message [ 4487.204268] Lustre: lustre-MDT0001: new disk, initializing [ 4487.276488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4487.311872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4487.322624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4492.023615] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 4501.297062] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4506.400744] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4513.436276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4513.665135] Lustre: lustre-OST0000: new disk, initializing [ 4513.668967] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4513.674143] Lustre: 161356:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4513.765627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4520.402808] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 4521.069908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4521.087774] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4521.243304] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4529.878484] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4533.367376] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4535.465623] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 14:57:16 (1787857036) [ 4784.095886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4784.102655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4784.118214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4785.636119] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4785.636689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4785.654843] Lustre: Skipped 1 previous similar message [ 4785.684145] Lustre: Skipped 2 previous similar messages [ 4789.961938] Lustre: server umount lustre-MDT0000 complete [ 4790.752895] LustreError: 160426:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4790.775331] LustreError: 160426:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4793.220705] LustreError: 159378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857295 with bad export cookie 11604999980018206328 [ 4793.225326] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4793.229161] LustreError: 159378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4795.875864] LustreError: 163106:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4795.876493] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4795.908556] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4798.879805] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4799.532693] Lustre: server umount lustre-MDT0001 complete [ 4817.375428] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4817.619533] Lustre: server umount lustre-OST0000 complete [ 4827.556405] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_hostid [ 4837.058656] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 4872.299846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4872.639815] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4872.671570] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4872.745086] Lustre: lustre-MDT0000: new disk, initializing [ 4872.825467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4872.846206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4877.176242] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 4888.462797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4888.626783] Lustre: 166031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4888.701363] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4888.706856] Lustre: Skipped 1 previous similar message [ 4888.792950] Lustre: lustre-MDT0001: new disk, initializing [ 4888.869477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4888.927734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4888.941826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4893.013367] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 4903.043495] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4910.361846] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4917.657527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4918.024761] Lustre: lustre-OST0000: new disk, initializing [ 4918.028604] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4918.041803] Lustre: 167999:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4918.160433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4924.558692] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 4927.523793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4927.536021] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4927.663760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4933.744903] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4938.618682] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4938.883951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4943.820437] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4944.031075] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4947.949656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4947.965071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4947.978455] Lustre: Skipped 1 previous similar message [ 4947.978735] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4948.030279] Lustre: Skipped 1 previous similar message [ 4958.178655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4958.189385] Lustre: Skipped 3 previous similar messages [ 4960.223434] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4960.349260] Lustre: server umount lustre-OST0000 complete [ 4965.856246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4965.866113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4969.971318] Lustre: server umount lustre-MDT0000 complete [ 4973.501695] LustreError: 166023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857476 with bad export cookie 11604999980018365158 [ 4973.505854] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4973.514086] LustreError: 166023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4973.787663] Lustre: server umount lustre-MDT0001 complete [ 4981.822410] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:04:42 (1787857482) [ 4997.915809] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_hostid [ 5008.009096] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 5050.690889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5050.903849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5050.930266] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5050.995742] Lustre: lustre-MDT0000: new disk, initializing [ 5051.124428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5051.139934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5055.530778] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 5066.319585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5066.449761] Lustre: 172756:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5066.706250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5071.699948] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 5081.987948] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5088.854555] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5097.856485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5098.329975] Lustre: lustre-OST0000: new disk, initializing [ 5098.332626] Lustre: Skipped 1 previous similar message [ 5098.337507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5098.341122] Lustre: Skipped 2 previous similar messages [ 5098.346273] Lustre: 174732:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5104.669869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5104.678982] Lustre: Skipped 1 previous similar message [ 5104.685532] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5104.761508] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5105.402686] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 5114.831091] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5125.095389] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5125.098481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5125.105562] Lustre: Skipped 1 previous similar message [ 5125.112055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5125.117345] Lustre: Skipped 3 previous similar messages [ 5129.191328] Lustre: server umount lustre-OST0000 complete [ 5133.283823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5133.293755] LustreError: Skipped 1 previous similar message [ 5147.103129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5147.357781] Lustre: server umount lustre-MDT0000 complete [ 5151.038780] LustreError: 172749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857653 with bad export cookie 11604999980018366040 [ 5151.047370] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5151.054393] LustreError: 172749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5151.455426] Lustre: server umount lustre-MDT0001 complete [ 5160.359714] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:07:41 (1787857661) [ 5175.979091] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_hostid [ 5186.242347] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 5222.309400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5222.532780] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5222.555430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5222.649046] Lustre: lustre-MDT0000: new disk, initializing [ 5222.713744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5222.718188] Lustre: Skipped 2 previous similar messages [ 5222.730097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5226.976814] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 5237.887348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5238.017284] Lustre: 179060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5238.281082] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5242.850462] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 5251.846448] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5257.437371] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5264.984571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5265.257878] Lustre: 181035:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5271.520718] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 5271.647531] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5271.789832] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5281.785172] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5292.512403] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5292.515380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5292.525465] LustreError: Skipped 1 previous similar message [ 5292.526813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5292.550704] Lustre: Skipped 4 previous similar messages [ 5292.552163] Lustre: Skipped 8 previous similar messages [ 5296.194557] Lustre: server umount lustre-OST0000 complete [ 5306.517593] Lustre: server umount lustre-MDT0000 complete [ 5307.872838] LustreError: 179069:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5307.909400] LustreError: 179069:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5310.377848] LustreError: 179050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857813 with bad export cookie 11604999980018367125 [ 5310.379373] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5310.393938] LustreError: 179050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5310.901897] Lustre: server umount lustre-MDT0001 complete [ 5322.298101] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 5324.718305] Key type lgssc unregistered [ 5324.979075] LNet: 182895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5324.983546] LNetError: 182895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5324.999700] LNet: Removed LNI 192.168.206.132@tcp [ 5325.955084] Key type .llcrypt unregistered [ 5325.961170] Key type ._llcrypt unregistered [ 5348.400509] Key type ._llcrypt registered [ 5348.415734] Key type .llcrypt registered [ 5348.521416] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 5349.550471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5349.653770] alg: No test for adler32 (adler32-zlib) [ 5350.694991] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 5350.876928] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 5352.543151] Key type lgssc registered [ 5353.412950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5359.783520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5359.814195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5361.215974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5365.174877] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 5370.341939] LustreError: 184312:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5370.356543] LustreError: 184312:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5373.782184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5374.186178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5378.998075] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 5387.413673] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5393.420878] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5400.708979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5401.131988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5406.248386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5407.868577] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 5415.318551] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5436.905507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5436.911528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5436.929803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5438.965743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5438.968925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5438.997929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5442.893183] Lustre: server umount lustre-OST0000 complete [ 5449.184823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5449.188250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5449.209367] Lustre: Skipped 1 previous similar message [ 5449.235759] Lustre: Skipped 2 previous similar messages [ 5453.569472] Lustre: server umount lustre-MDT0000 complete [ 5454.313575] LustreError: 184313:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5454.329449] LustreError: 184313:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5457.958919] LustreError: 184293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857960 with bad export cookie 11757798685919062279 [ 5457.980555] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5457.986139] LustreError: 184293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5458.718314] Lustre: server umount lustre-MDT0001 complete [ 5471.138680] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 5474.631284] Key type lgssc unregistered [ 5474.945235] LNet: 187870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5474.954246] LNetError: 187870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5474.985818] LNet: Removed LNI 192.168.206.132@tcp [ 5475.886751] Key type .llcrypt unregistered [ 5475.890570] Key type ._llcrypt unregistered [ 5500.963474] Key type ._llcrypt registered [ 5500.968265] Key type .llcrypt registered [ 5501.065471] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 5502.301604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5502.354098] alg: No test for adler32 (adler32-zlib) [ 5503.528178] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 5503.896307] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 5505.551461] Key type lgssc registered [ 5506.836499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5514.730762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5514.744984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5516.257645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5520.642108] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 5525.537757] LustreError: 189289:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5525.554452] LustreError: 189289:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5530.351138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5530.542556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5535.657159] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 5543.794887] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5550.968672] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5558.197215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5558.569509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5560.627663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5565.485205] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 5573.044947] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5583.757341] Lustre: server umount lustre-OST0000 complete [ 5590.885220] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff892847404800.ost_server_uuid 50 [ 5602.767472] Lustre: 188600:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858089/real 1787858089] req@0000000097c14a2b x1874704996526720/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787858105 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5602.771786] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5602.796290] Lustre: 188600:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5602.815918] Lustre: Skipped 1 previous similar message [ 5603.696929] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5603.874856] Lustre: 188601:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858089/real 1787858089] req@00000000ebab9e72 x1874704996527104/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787858105 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5603.902096] Lustre: 188601:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5607.945142] Lustre: 188599:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858094/real 1787858094] req@0000000007d38377 x1874704996527744/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787858110 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5614.047962] Lustre: 188600:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858099/real 1787858099] req@0000000053bfcdb8 x1874704996529280/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787858115 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5614.080764] Lustre: 188600:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5617.632450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5617.643612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5617.662127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5618.720848] Lustre: 188599:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858104/real 1787858104] req@000000000d7cfe13 x1874704996530432/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787858120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5618.751985] Lustre: 188599:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5623.264785] LustreError: 189289:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5623.275601] LustreError: 189289:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5623.306995] Lustre: server umount lustre-MDT0000 complete [ 5626.878717] LustreError: 189269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858129 with bad export cookie 15360488424495959224 [ 5626.889458] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5626.896257] LustreError: 189269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5627.244836] Lustre: server umount lustre-MDT0001 complete [ 5635.794692] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 15:15:36 (1787858136) [ 5642.341942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5642.693939] LustreError: 193223:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5642.758949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5646.815947] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 5647.841473] LustreError: 193224:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5652.970066] LustreError: 193223:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5655.335078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5655.765133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5660.084262] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 5667.868081] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5673.231075] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5680.314224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5680.566443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5686.234707] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 5687.805291] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5694.043213] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5700.894989] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5701.171380] Lustre: lustre-OST0001: new disk, initializing [ 5701.178889] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5701.188990] Lustre: 196063:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5701.290912] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5708.266571] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 5710.951307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5710.959917] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5711.071881] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5717.454205] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5721.575502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5721.579660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5721.584401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5721.598464] Lustre: Skipped 2 previous similar messages [ 5721.603431] Lustre: Skipped 2 previous similar messages [ 5722.958516] Lustre: lustre-OST0000: Not available for connect from 192.168.206.32@tcp (stopping) [ 5722.970257] Lustre: Skipped 1 previous similar message [ 5726.696458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5731.815473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5731.830699] Lustre: Skipped 3 previous similar messages [ 5734.367163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5734.558635] Lustre: server umount lustre-OST0000 complete [ 5736.938487] LustreError: 195047:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5736.953647] LustreError: 195047:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5741.185647] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5741.401890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5746.500161] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5746.738196] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5753.656858] LustreError: 195045:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5753.676440] LustreError: 195045:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 9 previous similar messages [ 5762.528815] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5762.546978] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5762.569189] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5762.574644] Lustre: Skipped 2 previous similar messages [ 5767.261597] Lustre: server umount lustre-OST0001 complete [ 5772.770657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5772.787635] Lustre: Skipped 1 previous similar message [ 5777.743483] Lustre: server umount lustre-MDT0000 complete [ 5781.357920] LustreError: 196062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858284 with bad export cookie 15360488424495960295 [ 5781.364447] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5781.388713] LustreError: 196062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5781.789960] Lustre: server umount lustre-MDT0001 complete [ 5805.413837] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 15:18:26 (1787858306) [ 5811.443996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5811.675173] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5811.718304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5811.906497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5815.989682] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 5823.970836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5824.080055] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5824.134395] Lustre: 199502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5828.772880] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 5836.067144] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5841.034948] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5848.175386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5848.400714] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5854.311370] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 5857.403377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5861.464921] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5868.751582] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5868.824776] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5875.556123] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 5883.606988] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5887.968132] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5887.979040] LustreError: Skipped 1 previous similar message [ 5887.983823] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5888.002214] Lustre: Skipped 1 previous similar message [ 5888.015218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5888.025131] Lustre: Skipped 2 previous similar messages [ 5892.468271] Lustre: server umount lustre-OST0000 complete [ 5894.456560] LustreError: 201353:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5894.486329] LustreError: 201353:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 5910.504301] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5910.507703] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5910.512931] LustreError: Skipped 1 previous similar message [ 5910.554555] Lustre: Skipped 2 previous similar messages [ 5914.040523] Lustre: server umount lustre-OST0001 complete [ 5920.741885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5920.758685] Lustre: Skipped 7 previous similar messages [ 5923.692981] Lustre: server umount lustre-MDT0000 complete [ 5926.796078] LustreError: 201374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858429 with bad export cookie 15360488424495961702 [ 5926.805410] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5926.808154] LustreError: 201374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5927.140733] Lustre: server umount lustre-MDT0001 complete [ 5954.223557] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 15:20:54 (1787858454) [ 5973.228413] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_hostid [ 5983.771318] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 6021.368065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6021.664744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6021.709608] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6021.817288] Lustre: lustre-MDT0000: new disk, initializing [ 6021.938098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6021.944038] Lustre: Skipped 3 previous similar messages [ 6021.965204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6025.748492] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 6036.132510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6036.251617] Lustre: 207600:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6036.309694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6036.312930] Lustre: Skipped 1 previous similar message [ 6036.367879] Lustre: lustre-MDT0001: new disk, initializing [ 6036.502416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6036.515141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6040.731463] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 6049.867173] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6055.784369] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6063.055917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6063.287379] Lustre: lustre-OST0000: new disk, initializing [ 6063.290918] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6063.298182] Lustre: 209566:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6069.449185] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 6070.346450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6070.359893] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6070.488557] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6078.981719] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6084.269326] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6084.540922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6090.558214] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6090.803375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6096.370713] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6096.377874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6096.380647] LustreError: Skipped 1 previous similar message [ 6096.392705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6096.410026] Lustre: Skipped 3 previous similar messages [ 6096.425614] Lustre: Skipped 1 previous similar message [ 6107.103464] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6107.376940] Lustre: server umount lustre-OST0000 complete [ 6117.394754] Lustre: server umount lustre-MDT0000 complete [ 6120.951704] LustreError: 207590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858623 with bad export cookie 15360488424495963263 [ 6120.963939] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6120.972213] LustreError: 207590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6121.484482] Lustre: server umount lustre-MDT0001 complete [ 6129.619269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6130.066161] LustreError: 211786:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6130.087943] LustreError: 211786:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 12 previous similar messages [ 6134.892468] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 6143.693847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6148.529645] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 6155.126586] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6160.519590] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6167.048664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6167.562857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6167.581521] Lustre: Skipped 4 previous similar messages [ 6173.741201] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 6181.460114] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6186.226774] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6186.439316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6190.629263] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6190.881308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6194.144655] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6194.158614] LustreError: Skipped 1 previous similar message [ 6194.167928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6194.182197] Lustre: Skipped 2 previous similar messages [ 6199.015231] Lustre: server umount lustre-OST0000 complete [ 6205.789287] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6211.414640] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6240.765323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6246.333751] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6246.352120] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6248.119406] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 6255.839341] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6258.376306] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6260.710558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.730057] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6265.827683] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6265.830110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6265.834933] LustreError: Skipped 1 previous similar message [ 6265.854881] Lustre: Skipped 12 previous similar messages [ 6268.330336] Lustre: server umount lustre-OST0000 complete [ 6278.291949] Lustre: server umount lustre-MDT0000 complete [ 6281.197916] LustreError: 211787:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6281.235173] LustreError: 211787:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 6282.898070] LustreError: 211768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858785 with bad export cookie 15360488424495964145 [ 6282.907852] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6282.915140] LustreError: 211768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6283.283345] Lustre: server umount lustre-MDT0001 complete [ 6293.866681] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:26:34 (1787858794) [ 6301.384062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6306.673050] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 6316.157812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6321.328148] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 6329.223882] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6335.933231] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6343.702097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6351.026716] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 6358.900372] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6364.597449] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6364.852709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6369.670297] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6369.885398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6376.805724] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6377.000361] Lustre: lustre-OST0001: new disk, initializing [ 6377.013586] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6377.024688] Lustre: 221126:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6382.507510] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 6385.286942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6385.296246] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6385.386070] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6391.112573] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6395.063861] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6395.241827] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6398.893696] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6399.152556] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6400.994458] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6400.997678] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6401.010164] LustreError: Skipped 1 previous similar message [ 6401.045439] Lustre: Skipped 6 previous similar messages [ 6415.327149] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6415.467033] Lustre: server umount lustre-OST0001 complete [ 6422.914898] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6423.230412] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6428.186824] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6428.386400] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6457.440854] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6457.583102] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6457.591269] Lustre: Skipped 5 previous similar messages [ 6462.956736] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6462.975375] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6462.978810] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6463.006860] Lustre: Skipped 1 previous similar message [ 6463.338204] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 6470.854734] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6472.154624] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6480.401227] Lustre: server umount lustre-OST0001 complete [ 6511.071248] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6511.241330] Lustre: server umount lustre-OST0000 complete [ 6521.291148] Lustre: server umount lustre-MDT0000 complete [ 6524.761095] LustreError: 217739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859027 with bad export cookie 15360488424495965111 [ 6524.767766] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6524.772939] LustreError: 217739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6539.231153] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6539.587540] Lustre: server umount lustre-MDT0001 complete [ 6564.986290] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 15:31:05 (1787859065) [ 6571.609528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6572.003236] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6572.062376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6576.905340] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 6585.731577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6585.822870] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6585.853442] Lustre: 226375:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6590.387833] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 6597.250960] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6602.419490] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6609.032442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6609.209926] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6615.577761] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 6622.779702] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6632.259126] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6632.304260] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6639.031438] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 6646.746477] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6648.471466] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6653.496540] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6653.749215] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6658.795517] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6659.007486] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6665.126031] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8928509cb800.ost_server_uuid 50 [ 6666.697477] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8928509cb800.ost_server_uuid in IDLE state after 0 sec [ 6667.991594] Lustre: Permanently deactivating lustre-OST0001 [ 6668.030307] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6672.373299] Lustre: Permanently reactivating lustre-OST0001 [ 6672.393170] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6672.399277] Lustre: Skipped 2 previous similar messages [ 6677.994277] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6678.002460] LustreError: Skipped 3 previous similar messages [ 6678.008458] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6678.015784] Lustre: Skipped 6 previous similar messages [ 6678.021453] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6678.029480] Lustre: Skipped 21 previous similar messages [ 6691.297788] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6691.451756] Lustre: server umount lustre-OST0001 complete [ 6694.379670] LustreError: 228222:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6694.397294] LustreError: 228222:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 42 previous similar messages [ 6701.951857] Lustre: server umount lustre-OST0000 complete [ 6712.335164] Lustre: server umount lustre-MDT0000 complete [ 6717.440253] LustreError: 226368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859220 with bad export cookie 15360488424495966399 [ 6717.445359] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6717.453205] LustreError: 226368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6718.210896] Lustre: server umount lustre-MDT0001 complete [ 6745.159164] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 15:34:05 (1787859245) [ 6754.242489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6754.844799] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6754.882681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6754.899185] Lustre: Skipped 2 previous similar messages [ 6760.550460] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 6769.525817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6769.671989] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6769.703682] Lustre: 233072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6775.804281] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 6785.427199] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6792.720833] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6801.457554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6801.694714] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6801.718529] Lustre: Permanently deactivating lustre-OST0000 [ 6801.735448] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6807.129639] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6809.251631] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 6818.336895] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6825.796133] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6825.877707] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6834.050278] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 6836.455602] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6843.060759] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6845.142310] Lustre: Permanently reactivating lustre-OST0000 [ 6845.178908] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6845.189313] Lustre: Skipped 2 previous similar messages [ 6854.235725] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6854.236976] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 47 seconds. I think it's dead, and I am evicting it. exp 000000002e47470d, cur 1787859356 deadline 1787859339 last 1787859309 [ 6854.247458] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6854.248463] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6854.347345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6868.590572] Lustre: server umount lustre-OST0001 complete [ 6878.984664] Lustre: server umount lustre-OST0000 complete [ 6889.812094] Lustre: server umount lustre-MDT0000 complete [ 6894.131843] LustreError: 233063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859396 with bad export cookie 15360488424495968030 [ 6894.142072] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6894.147708] LustreError: 233063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6894.545542] Lustre: server umount lustre-MDT0001 complete [ 6907.732964] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 6911.400869] Key type lgssc unregistered [ 6911.731114] LNet: 237962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6911.747346] LNetError: 237962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6911.763201] LNet: Removed LNI 192.168.206.132@tcp [ 6912.892601] Key type .llcrypt unregistered [ 6912.897789] Key type ._llcrypt unregistered [ 6940.130510] Key type ._llcrypt registered [ 6940.133552] Key type .llcrypt registered [ 6940.278532] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 6942.184172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6942.250092] alg: No test for adler32 (adler32-zlib) [ 6943.459882] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 6943.807120] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 6945.511766] Key type lgssc registered [ 6947.107574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6958.482350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6958.508629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6960.197858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6965.948490] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 6969.315438] LustreError: 239430:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6969.357898] LustreError: 239430:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6974.452637] LustreError: 239431:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6976.336881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6976.573843] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6976.631402] Lustre: 239418:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6976.642752] Lustre: Permanently deactivating lustre-MDT0001 [ 6976.655906] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6977.090965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6983.076221] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 6983.181804] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6990.593530] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6996.176379] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7002.775483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7003.086769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7005.164787] LustreError: 241274:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7005.197674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7009.776213] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 7010.274959] LustreError: 241273:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7015.402468] LustreError: 241395:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7019.399508] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7021.592283] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7026.663412] LustreError: 241275:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7026.671210] LustreError: 241275:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 7028.572460] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7029.000551] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7034.369121] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7034.371622] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7036.060862] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 7045.141683] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7049.415403] Lustre: Permanently reactivating lustre-MDT0001 [ 7049.457968] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7049.462856] Lustre: Skipped 2 previous similar messages [ 7052.116036] Lustre: lustre-MDT0001: Client 814c7e37-7704-465a-800c-f5fbd1a52a7d (at 192.168.206.32@tcp) reconnecting [ 7058.635649] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7058.660984] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7058.671568] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7058.689555] LustreError: 243143:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7058.694181] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7058.700776] LustreError: 243143:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7058.700941] LustreError: 243143:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.132@tcp: cfg command failed: rc = -108 [ 7058.702089] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7058.702089] [ 7058.711695] LustreError: 239422:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7060.108143] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7062.285351] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7069.434063] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7069.816273] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7075.800895] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7076.088191] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7079.294889] Lustre: Permanently deactivating lustre-MDT0001 [ 7079.320106] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7079.325392] Lustre: Skipped 2 previous similar messages [ 7084.675197] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7099.877796] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7099.889515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7099.890357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7099.894113] Lustre: Skipped 3 previous similar messages [ 7099.931573] Lustre: Skipped 3 previous similar messages [ 7103.390282] Lustre: server umount lustre-MDT0000 complete [ 7104.994330] LustreError: 239427:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7105.014500] LustreError: 239427:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 7108.101729] LustreError: 239410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859610 with bad export cookie 3603757874190409123 [ 7108.106622] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7108.114846] LustreError: 239410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7108.459200] Lustre: server umount lustre-MDT0001 complete [ 7115.059826] Lustre: server umount lustre-OST0000 complete [ 7129.243697] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 15:40:29 (1787859629) [ 7141.040651] Lustre: server umount lustre-OST0001 complete [ 7149.002491] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_hostid [ 7158.470195] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 7195.256719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7195.612672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7195.632026] Lustre: Skipped 2 previous similar messages [ 7195.674769] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7195.750874] Lustre: lustre-MDT0000: new disk, initializing [ 7195.815675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7195.837653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7200.848244] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 7211.708493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7211.839534] Lustre: 247714:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7211.896178] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7211.901752] Lustre: Skipped 1 previous similar message [ 7211.975371] Lustre: lustre-MDT0001: new disk, initializing [ 7212.076294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7212.108569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7212.120654] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7216.766476] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 7226.485831] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7232.051593] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7238.605061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7238.818501] Lustre: lustre-OST0000: new disk, initializing [ 7238.822843] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7238.827224] Lustre: 249682:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7238.903485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7244.370244] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 7248.475863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7248.492323] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7248.630374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7254.943195] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7257.327274] LustreError: 247723:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7266.015692] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7266.272985] Lustre: lustre-OST0001: new disk, initializing [ 7266.276841] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7266.286497] Lustre: 250799:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7267.333846] LustreError: 247723:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7267.375135] LustreError: 247722:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7271.630236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7271.641016] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7271.747142] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7272.740313] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 7277.384730] LustreError: 247722:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7283.889931] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7285.871155] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7292.384329] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7292.396067] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7292.414776] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7292.424315] Lustre: Skipped 1 previous similar message [ 7294.279518] Lustre: server umount lustre-OST0001 complete [ 7297.521751] LustreError: 249667:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7297.538381] LustreError: 249667:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7315.433820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7315.439482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7315.441200] LustreError: Skipped 2 previous similar messages [ 7315.442414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7315.461392] Lustre: Skipped 2 previous similar messages [ 7317.996124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7318.004971] Lustre: Skipped 2 previous similar messages [ 7323.113891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7323.121580] Lustre: Skipped 1 previous similar message [ 7327.199164] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7327.475664] Lustre: server umount lustre-OST0000 complete [ 7333.355428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7333.369418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7333.376732] Lustre: Skipped 1 previous similar message [ 7337.854282] Lustre: server umount lustre-MDT0000 complete [ 7341.769591] LustreError: 249681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859844 with bad export cookie 3603757874190411930 [ 7341.770747] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7341.790153] LustreError: 249681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7356.383605] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7356.648042] Lustre: server umount lustre-MDT0001 complete [ 7367.613686] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 7370.411965] Key type lgssc unregistered [ 7370.747757] LNet: 252836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7370.766435] LNetError: 252836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7370.790643] LNet: Removed LNI 192.168.206.132@tcp [ 7371.678929] Key type .llcrypt unregistered [ 7371.680704] Key type ._llcrypt unregistered [ 7416.104822] Key type ._llcrypt registered [ 7416.107426] Key type .llcrypt registered [ 7416.264690] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 7417.867860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7417.893490] alg: No test for adler32 (adler32-zlib) [ 7418.950728] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 7419.147614] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 7420.799211] Key type lgssc registered [ 7422.207355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7430.344551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7430.366206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7431.567930] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7431.604457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7431.729335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7436.145704] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 7445.370740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7445.550225] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7445.593586] Lustre: 255046:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7445.918448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7451.143901] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 7459.810687] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7466.756757] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7473.292101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7473.577018] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7473.789555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7480.586334] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 7484.077659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7489.754821] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7543.780098] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7543.794198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7543.802043] Lustre: Skipped 1 previous similar message [ 7548.655222] Lustre: server umount lustre-OST0000 complete [ 7552.782747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7565.278812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7565.708369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7572.176355] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 7580.288803] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7584.108505] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7584.129313] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7584.135930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7625.186087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7625.189361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7625.192789] LustreError: Skipped 1 previous similar message [ 7625.195922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7625.232840] Lustre: Skipped 2 previous similar messages [ 7626.907921] Lustre: server umount lustre-OST0000 complete [ 7635.445155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7635.446417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7635.465374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7635.484873] Lustre: Skipped 2 previous similar messages [ 7636.779511] Lustre: server umount lustre-MDT0000 complete [ 7640.043289] LustreError: 255038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860142 with bad export cookie 15493174147812751379 [ 7640.049500] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7640.058035] LustreError: 255038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7640.376700] Lustre: server umount lustre-MDT0001 complete [ 7649.314873] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 7651.652647] Key type lgssc unregistered [ 7651.948620] LNet: 260473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7651.955311] LNetError: 260473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7653.030064] LNet: Removed LNI 192.168.206.132@tcp [ 7653.853291] Key type .llcrypt unregistered [ 7653.857388] Key type ._llcrypt unregistered [ 7675.877750] Key type ._llcrypt registered [ 7675.880280] Key type .llcrypt registered [ 7676.034718] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 7677.059108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7677.076349] alg: No test for adler32 (adler32-zlib) [ 7678.103228] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 7678.426085] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 7680.199197] Key type lgssc registered [ 7681.610205] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7691.032075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7691.043018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7692.557042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7697.764903] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 7703.010553] LustreError: 261890:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7703.026602] LustreError: 261890:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7705.622606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7705.923806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7709.511823] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 7718.545991] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7724.148586] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7731.705638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7732.048882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7738.227178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7738.581656] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 7745.788260] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7774.176844] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7774.190144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7774.220431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7776.224737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7776.227563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7776.235550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7776.251563] Lustre: Skipped 1 previous similar message [ 7778.131144] Lustre: server umount lustre-OST0000 complete [ 7782.882685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7782.904987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7782.941383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7786.465662] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7786.468663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7786.485739] Lustre: Skipped 1 previous similar message [ 7788.625466] Lustre: server umount lustre-MDT0000 complete [ 7791.584592] LustreError: 264382:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7791.599595] LustreError: 264382:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7792.140816] LustreError: 262626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860294 with bad export cookie 2942284690171781686 [ 7792.154647] LustreError: 262626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7792.160875] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7792.517359] Lustre: server umount lustre-MDT0001 complete [ 7802.839872] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 7805.469463] Key type lgssc unregistered [ 7805.810166] LNet: 266203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7805.824927] LNetError: 266203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7805.847620] LNet: Removed LNI 192.168.206.132@tcp [ 7806.620918] Key type .llcrypt unregistered [ 7806.628885] Key type ._llcrypt unregistered [ 7824.342325] Key type ._llcrypt registered [ 7824.344500] Key type .llcrypt registered [ 7824.447487] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 7825.184211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7825.207204] alg: No test for adler32 (adler32-zlib) [ 7826.352990] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 7826.655191] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 7828.359145] Key type lgssc registered [ 7829.546965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7837.018163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7837.041160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7838.466287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7842.790588] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 7848.939246] LustreError: 267429:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7848.971710] LustreError: 267429:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7850.962359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7851.282798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7855.656917] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 7863.698973] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7870.197392] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7877.506645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7877.892125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7884.017161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7885.150766] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 7892.918863] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7911.915331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7911.915383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7911.927339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7911.954030] Lustre: Skipped 1 previous similar message [ 7916.290363] Lustre: server umount lustre-OST0000 complete [ 7922.145200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7922.146683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7922.155831] Lustre: Skipped 1 previous similar message [ 7922.168898] Lustre: Skipped 2 previous similar messages [ 7926.542886] Lustre: server umount lustre-MDT0000 complete [ 7927.266824] LustreError: 267425:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7927.277398] LustreError: 267425:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7930.226204] LustreError: 267410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860432 with bad export cookie 5709001283795445541 [ 7930.232517] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7930.240072] LustreError: 267410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7930.610415] Lustre: server umount lustre-MDT0001 complete [ 7941.541510] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 7944.165857] Key type lgssc unregistered [ 7944.424327] LNet: 271110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7944.435559] LNetError: 271110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7944.451832] LNet: Removed LNI 192.168.206.132@tcp [ 7945.305136] Key type .llcrypt unregistered [ 7945.307066] Key type ._llcrypt unregistered [ 7968.038878] Key type ._llcrypt registered [ 7968.042189] Key type .llcrypt registered [ 7968.206871] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 7969.378854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7969.413071] alg: No test for adler32 (adler32-zlib) [ 7970.478161] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 7970.731219] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 7972.391222] Key type lgssc registered [ 7973.523730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7982.112353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7982.127661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7983.726362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7988.712979] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 7993.313257] LustreError: 272533:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7993.361205] LustreError: 272533:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7998.453691] LustreError: 272534:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7999.198289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7999.621983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8003.855928] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 8011.276124] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8017.903812] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8025.039615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8025.628965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8028.729195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8032.108696] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 8039.732941] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8065.513844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8065.524509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8065.545582] Lustre: Skipped 1 previous similar message [ 8065.568604] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8070.626870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8070.640239] Lustre: Skipped 2 previous similar messages [ 8071.406146] Lustre: server umount lustre-OST0000 complete [ 8075.748044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8075.751916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8075.774671] Lustre: Skipped 1 previous similar message [ 8080.864554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8080.870540] Lustre: Skipped 2 previous similar messages [ 8082.184929] Lustre: server umount lustre-MDT0000 complete [ 8085.988829] LustreError: 272534:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8086.019368] LustreError: 272534:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8086.451062] LustreError: 272513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860589 with bad export cookie 17357749026352465004 [ 8086.456167] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8086.461260] LustreError: 272513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8086.974855] Lustre: server umount lustre-MDT0001 complete [ 8098.801685] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 8101.950534] Key type lgssc unregistered [ 8102.263032] LNet: 276841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8102.267237] LNetError: 276841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8102.282655] LNet: Removed LNI 192.168.206.132@tcp [ 8103.212296] Key type .llcrypt unregistered [ 8103.218369] Key type ._llcrypt unregistered [ 8119.660462] Key type ._llcrypt registered [ 8119.662754] Key type .llcrypt registered [ 8119.778234] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 8121.032182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8121.049950] alg: No test for adler32 (adler32-zlib) [ 8122.188926] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 8122.430089] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 8124.087227] Key type lgssc registered [ 8125.158669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8132.795565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8132.826563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8134.357437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8138.802230] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 8144.871970] LustreError: 278079:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8144.904855] LustreError: 278079:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8147.642205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8148.080594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8153.075281] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 8161.680916] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8167.901570] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8176.171166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8176.657791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8179.782087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8184.140360] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 8193.217187] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8203.235688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8203.236727] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8203.251312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8203.268665] Lustre: Skipped 1 previous similar message [ 8208.356665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8208.369279] Lustre: Skipped 1 previous similar message [ 8209.326276] Lustre: server umount lustre-OST0000 complete [ 8213.478712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8213.496672] Lustre: Skipped 1 previous similar message [ 8213.501907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8213.506282] Lustre: Skipped 1 previous similar message [ 8218.595335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8218.605823] Lustre: Skipped 1 previous similar message [ 8223.719238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8223.741750] Lustre: Skipped 1 previous similar message [ 8227.296510] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8227.604107] Lustre: server umount lustre-MDT0000 complete [ 8228.842460] LustreError: 278083:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8228.871285] LustreError: 278083:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8232.194274] LustreError: 278044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860734 with bad export cookie 1888655470127041640 [ 8232.201574] LustreError: 278044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8232.203809] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8232.444300] Lustre: server umount lustre-MDT0001 complete [ 8242.582540] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 8245.269900] Key type lgssc unregistered [ 8245.576524] LNet: 281784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8245.582053] LNetError: 281784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8245.604241] LNet: Removed LNI 192.168.206.132@tcp [ 8246.273573] Key type .llcrypt unregistered [ 8246.275370] Key type ._llcrypt unregistered [ 8258.663456] Key type ._llcrypt registered [ 8258.666193] Key type .llcrypt registered [ 8258.790745] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8276.159632] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_hostid [ 8291.758695] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 8293.145567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8293.351574] alg: No test for adler32 (adler32-zlib) [ 8294.552740] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 8294.877771] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 8296.672280] Key type lgssc registered [ 8298.038247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8340.224069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8340.258989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8341.587816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8341.636653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8341.780716] Lustre: lustre-MDT0000: new disk, initializing [ 8341.892329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8341.931922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8346.620359] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 8359.085657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8359.206899] Lustre: 285341:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8359.317719] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8359.321926] Lustre: Skipped 1 previous similar message [ 8359.424553] Lustre: lustre-MDT0001: new disk, initializing [ 8359.501633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8359.536212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8359.550100] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8365.400795] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 8376.197630] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8382.973865] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8390.203519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8390.624326] Lustre: lustre-OST0000: new disk, initializing [ 8390.635935] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8390.648305] Lustre: 287319:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8390.799901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8397.866079] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 8398.925706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8398.953329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8399.144778] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8408.469498] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8413.359462] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8413.631733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8419.396714] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8419.613321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8424.930391] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8424.935598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8424.940786] LustreError: Skipped 1 previous similar message [ 8424.955997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8424.972536] Lustre: Skipped 1 previous similar message [ 8427.722182] Lustre: server umount lustre-OST0000 complete [ 8435.169255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8435.186890] Lustre: Skipped 1 previous similar message [ 8435.197911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8435.203148] Lustre: Skipped 1 previous similar message [ 8437.840727] Lustre: server umount lustre-MDT0000 complete [ 8441.892804] LustreError: 285330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860944 with bad export cookie 3545119741072265953 [ 8441.900187] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8441.905438] LustreError: 285330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8442.285579] Lustre: server umount lustre-MDT0001 complete [ 8451.464595] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 16:02:32 (1787860952) [ 8458.215436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8458.624170] LustreError: 289734:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8458.648828] LustreError: 289734:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 8458.728340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8462.714080] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 8463.840943] LustreError: 289735:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8468.960962] LustreError: 289734:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8470.459530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8470.754227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8474.684576] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 8482.278610] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8488.619144] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8495.334313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8495.662963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8501.911642] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 8509.190194] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8519.143441] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8519.145924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8519.154969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8519.168835] Lustre: Skipped 2 previous similar messages [ 8523.204571] Lustre: server umount lustre-OST0000 complete [ 8527.331204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8527.338600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8527.354410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8527.362763] Lustre: Skipped 1 previous similar message [ 8533.810149] Lustre: server umount lustre-MDT0000 complete [ 8534.497692] LustreError: 290976:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8534.521141] LustreError: 290976:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8538.094896] LustreError: 292518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861040 with bad export cookie 3545119741072266835 [ 8538.100524] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8538.110733] LustreError: 292518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8538.521964] Lustre: server umount lustre-MDT0001 complete [ 8550.413831] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 8553.594952] Key type lgssc unregistered [ 8553.994633] LNet: 293272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8554.005854] LNetError: 293272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8554.038320] LNet: Removed LNI 192.168.206.132@tcp [ 8554.973784] Key type .llcrypt unregistered [ 8554.975566] Key type ._llcrypt unregistered [ 8583.010142] Key type ._llcrypt registered [ 8583.014987] Key type .llcrypt registered [ 8583.165343] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_hostid [ 8598.882496] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 8599.920654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8600.096115] alg: No test for adler32 (adler32-zlib) [ 8601.133230] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 8601.435613] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 8603.151444] Key type lgssc registered [ 8604.328141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8648.654934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8648.689721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8650.043872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8650.119695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8650.233787] Lustre: lustre-MDT0000: new disk, initializing [ 8650.321526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8650.347667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8655.224396] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 8665.724271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8665.849901] Lustre: 297107:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8665.898458] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8665.902816] Lustre: Skipped 1 previous similar message [ 8665.964677] Lustre: lustre-MDT0001: new disk, initializing [ 8666.043354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8666.086521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8666.097657] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8670.213379] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 8679.418430] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8684.906875] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8691.999288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8692.386131] Lustre: lustre-OST2710: new disk, initializing [ 8692.390301] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8692.394829] Lustre: 299078:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8692.521352] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8699.592508] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 8700.494236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8700.508636] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8700.671352] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8709.929920] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8717.875212] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8718.144092] Lustre: lustre-OST03e8: new disk, initializing [ 8718.147671] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8718.155911] Lustre: 300140:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8718.242252] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8725.476682] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 8727.164966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8727.178908] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8727.328879] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8734.470189] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8743.482751] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8743.723871] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8748.538057] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8748.780654] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8753.844399] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8754.120863] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8759.171995] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8759.393385] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8766.180489] Lustre: server umount lustre-MDT0000 complete [ 8768.486836] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8768.492457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8768.492631] Lustre: Skipped 1 previous similar message [ 8769.476732] LustreError: 297098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861272 with bad export cookie 8588460674275328314 [ 8769.486877] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8769.490336] LustreError: 297098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8769.761842] Lustre: server umount lustre-MDT0001 complete [ 8783.955956] Lustre: server umount lustre-OST2710 complete [ 8798.150483] Lustre: server umount lustre-OST03e8 complete [ 8818.917157] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_hostid [ 8829.402069] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 8868.230766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8868.521521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8868.548913] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8868.611503] Lustre: lustre-MDT0000: new disk, initializing [ 8868.692483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8868.714848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8873.827124] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 8884.640621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8884.816741] Lustre: 305391:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8884.894987] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8884.901827] Lustre: Skipped 1 previous similar message [ 8885.040565] Lustre: lustre-MDT0001: new disk, initializing [ 8885.145598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8885.211881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8885.226698] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8889.351157] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 8898.684404] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8904.432707] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8910.536907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8910.852648] Lustre: lustre-OST0000: new disk, initializing [ 8910.863400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8910.872982] Lustre: 307362:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8910.961650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8917.644914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8917.657847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8917.865852] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8918.098821] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 8928.321974] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8933.376796] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8933.651935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8938.250364] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8938.534903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8943.593855] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8943.599553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8943.603961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8943.609345] Lustre: Skipped 3 previous similar messages [ 8943.640277] Lustre: Skipped 1 previous similar message [ 8946.680802] Lustre: server umount lustre-OST0000 complete [ 8951.776800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8951.784662] LustreError: Skipped 1 previous similar message [ 8951.792409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8951.812559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8953.829329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8953.835375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8957.057205] Lustre: server umount lustre-MDT0000 complete [ 8958.949079] LustreError: 305398:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8958.995027] LustreError: 305398:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 8961.321244] LustreError: 305382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861464 with bad export cookie 8588460674275329994 [ 8961.335307] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8961.339872] LustreError: 305382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8961.916459] Lustre: server umount lustre-MDT0001 complete [ 8972.205535] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 16:11:12 (1787861472) [ 8973.959420] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8976.210051] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 16:11:16 (1787861476) [ 8980.192247] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 9008.822831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9009.086856] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9009.141601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9009.306601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9013.755747] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 9022.262896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9022.374651] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9022.400287] Lustre: 311235:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9026.733340] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 9048.556227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9048.845676] Lustre: Denying initial registration attempt from nid 192.168.206.132@tcp, specified as failover [ 9048.852510] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9048.858637] LustreError: 312689:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9048.863554] LustreError: 312689:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9048.868578] LustreError: 312689:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9048.943919] Lustre: server umount lustre-OST0000 complete [ 9048.947663] LustreError: 312689:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9056.739341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9056.743377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9056.762378] Lustre: Skipped 1 previous similar message [ 9056.773903] Lustre: Skipped 2 previous similar messages [ 9061.865661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9061.881908] Lustre: Skipped 1 previous similar message [ 9062.511585] Lustre: server umount lustre-MDT0000 complete [ 9066.680149] LustreError: 311230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861569 with bad export cookie 8588460674275330876 [ 9066.693780] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9066.701367] LustreError: 311230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9067.042668] Lustre: server umount lustre-MDT0001 complete [ 9078.573566] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 9081.421757] Key type lgssc unregistered [ 9081.679511] LNet: 313813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9081.687101] LNetError: 313813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9081.704016] LNet: Removed LNI 192.168.206.132@tcp [ 9082.570496] Key type .llcrypt unregistered [ 9082.572963] Key type ._llcrypt unregistered [ 9094.824868] Key type ._llcrypt registered [ 9094.833441] Key type .llcrypt registered [ 9094.999147] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 9096.283230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9096.320890] alg: No test for adler32 (adler32-zlib) [ 9097.409377] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 9097.622794] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 9099.281678] Key type lgssc registered [ 9100.515935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9148.679040] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 9160.563341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9160.635431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9162.077991] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9162.088889] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9162.316301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9168.579531] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 9177.222120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9177.350177] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9177.391448] Lustre: 316581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9177.659925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9182.808816] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 9190.447564] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9196.059274] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9215.868369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9216.118603] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9216.316832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9222.539803] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 9230.002339] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9235.430596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9235.436097] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9235.437499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9235.441779] Lustre: Skipped 1 previous similar message [ 9238.172712] Lustre: server umount lustre-OST0000 complete [ 9244.130307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9244.137906] LustreError: Skipped 1 previous similar message [ 9244.144802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9244.156779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9244.162766] Lustre: Skipped 1 previous similar message [ 9245.664592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9245.666182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9245.683356] Lustre: Skipped 1 previous similar message [ 9247.817370] Lustre: server umount lustre-MDT0000 complete [ 9251.255077] LustreError: 318446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861753 with bad export cookie 2167142333850834098 [ 9251.258057] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9251.266437] LustreError: 318446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9251.564697] Lustre: server umount lustre-MDT0001 complete [ 9260.718591] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 9263.246918] Key type lgssc unregistered [ 9263.555074] LNet: 320161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9263.563236] LNetError: 320161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9263.578776] LNet: Removed LNI 192.168.206.132@tcp [ 9264.312696] Key type .llcrypt unregistered [ 9264.325462] Key type ._llcrypt unregistered [ 9288.681337] Key type ._llcrypt registered [ 9288.686181] Key type .llcrypt registered [ 9288.782973] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 9289.668656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9289.716104] alg: No test for adler32 (adler32-zlib) [ 9290.847760] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 9291.102952] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 9292.807207] Key type lgssc registered [ 9293.872880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9301.111478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9301.156815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9302.699299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9307.669454] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 9313.249373] LustreError: 321577:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9313.262648] LustreError: 321577:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 9316.439253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9316.750895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9321.055187] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 9328.192935] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9333.957106] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9342.090136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9342.559657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9348.528884] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 9355.892734] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9372.643212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9372.647694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9372.656025] Lustre: Skipped 2 previous similar messages [ 9372.668838] Lustre: Skipped 1 previous similar message [ 9375.911241] Lustre: server umount lustre-MDT0000 complete [ 9377.764341] LustreError: 322305:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9377.775611] LustreError: 322305:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 9379.167953] LustreError: 321559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861881 with bad export cookie 4261672124206864753 [ 9379.173560] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9379.185264] LustreError: 321559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9382.879915] LustreError: 321574:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9382.880189] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9382.893530] LustreError: 321574:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9382.920477] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9382.931666] Lustre: Skipped 2 previous similar messages [ 9384.933059] LustreError: 321574:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9384.933244] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9384.957958] LustreError: 321574:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9384.965908] Lustre: Skipped 1 previous similar message [ 9385.646735] Lustre: server umount lustre-MDT0001 complete [ 9391.965622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9400.891923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9403.363257] LustreError: 320887:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009c2d1a07 x1874708967502080/t0(0) o250->MGC192.168.206.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9403.626401] LustreError: 323410:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9403.744675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9403.804495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9407.624328] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 9413.990728] LustreError: 325190:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9413.994047] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9414.016904] LustreError: 325190:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 9416.310536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9416.703709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9420.994542] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 9421.810245] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9428.418830] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9430.079802] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9435.099459] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9436.635664] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9442.274949] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9442.277070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9442.286367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9442.299766] Lustre: Skipped 1 previous similar message [ 9442.310437] Lustre: Skipped 1 previous similar message [ 9447.398201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9447.410395] Lustre: Skipped 1 previous similar message [ 9456.095141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9456.448487] Lustre: server umount lustre-OST0000 complete [ 9462.757289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9462.757828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9462.784597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9462.798573] Lustre: Skipped 2 previous similar messages [ 9475.039151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9475.150845] Lustre: server umount lustre-MDT0000 complete [ 9478.113245] LustreError: 325191:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9478.140299] LustreError: 325191:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9479.223902] LustreError: 321559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861981 with bad export cookie 4261672124206872586 [ 9479.225496] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9479.239537] LustreError: 321559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9479.704635] Lustre: server umount lustre-MDT0001 complete [ 9492.099346] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 9495.348094] Key type lgssc unregistered [ 9495.809905] LNet: 327867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9495.817890] LNetError: 327867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9495.844785] LNet: Removed LNI 192.168.206.132@tcp [ 9496.735824] Key type .llcrypt unregistered [ 9496.739209] Key type ._llcrypt unregistered [ 9530.805051] Key type ._llcrypt registered [ 9530.806868] Key type .llcrypt registered [ 9530.905779] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 9532.150980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9532.158358] alg: No test for adler32 (adler32-zlib) [ 9533.132897] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 9533.358669] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 9535.023462] Key type lgssc registered [ 9536.232911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9544.638952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9544.675748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9546.107993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9551.466814] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 9555.450603] LustreError: 329290:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9555.478160] LustreError: 329290:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 9560.548689] LustreError: 329289:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9561.373841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9561.762541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9565.835505] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 9575.565654] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9583.123864] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9587.168755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9587.175990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9587.208255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9591.782773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9591.785409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9593.720756] Lustre: server umount lustre-MDT0000 complete [ 9596.899249] LustreError: 329285:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9596.914250] LustreError: 329285:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9597.624096] LustreError: 329271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862100 with bad export cookie 15642709426065772094 [ 9597.627666] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9597.646089] LustreError: 329271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9597.997748] Lustre: server umount lustre-MDT0001 complete [ 9613.167694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9613.393562] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9613.434127] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9613.589749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9618.057345] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 9627.231160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9627.406983] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9627.468331] Lustre: 331877:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9627.885405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9632.562174] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 9640.779285] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9647.316979] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9657.342582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9657.598195] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9657.605590] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9657.615456] LustreError: 333761:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9657.622065] LustreError: 333761:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9657.629724] LustreError: 333761:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9657.712448] Lustre: server umount lustre-OST0000 complete [ 9657.715929] LustreError: 333761:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9668.333559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9668.566979] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9668.845228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9674.402208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9676.548939] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 9685.363100] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9693.809709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9693.970456] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9694.141880] Lustre: lustre-OST0001: new disk, initializing [ 9694.157379] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9694.170102] Lustre: 335303:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9694.317473] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9700.538833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9700.559504] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9700.629471] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9701.278543] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 9708.938598] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9716.219385] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9716.222300] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9716.224291] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9716.224298] Lustre: Skipped 1 previous similar message [ 9716.236389] LustreError: Skipped 1 previous similar message [ 9716.298370] Lustre: Skipped 1 previous similar message [ 9718.310600] Lustre: server umount lustre-OST0001 complete [ 9721.331088] LustreError: 334227:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9721.354563] LustreError: 334227:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9725.409129] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9725.416512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9725.437700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9725.442363] Lustre: Skipped 1 previous similar message [ 9726.458480] LustreError: 334228:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9726.472542] LustreError: 334228:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9728.541133] Lustre: server umount lustre-OST0000 complete [ 9735.648517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9735.656333] LustreError: Skipped 1 previous similar message [ 9735.664183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9735.679690] Lustre: Skipped 1 previous similar message [ 9735.686454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9735.692140] Lustre: Skipped 3 previous similar messages [ 9739.392532] Lustre: server umount lustre-MDT0000 complete [ 9741.792801] LustreError: 333193:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9741.819105] LustreError: 333193:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9742.557263] LustreError: 331871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862245 with bad export cookie 15642709426065772647 [ 9742.566191] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9742.573844] LustreError: 331871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9742.882124] Lustre: server umount lustre-MDT0001 complete [ 9755.348711] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [ 9758.826211] Key type lgssc unregistered [ 9759.191385] LNet: 337264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9759.206085] LNetError: 337264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9759.228303] LNet: Removed LNI 192.168.206.132@tcp [ 9760.412151] Key type .llcrypt unregistered [ 9760.417089] Key type ._llcrypt unregistered [ 9841.405378] Key type ._llcrypt registered [ 9841.410416] Key type .llcrypt registered [ 9841.597079] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_hostid [ 9858.599217] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [ 9860.169686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9860.279804] alg: No test for adler32 (adler32-zlib) [ 9861.404695] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [ 9861.683309] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [ 9863.367170] Key type lgssc registered [ 9864.499727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9899.791468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9899.852897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9901.065925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9901.096610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9901.161894] Lustre: lustre-MDT0000: new disk, initializing [ 9901.297451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9901.330576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9906.216487] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 9917.553365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9917.725764] Lustre: 342204:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9917.789711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9917.797037] Lustre: Skipped 1 previous similar message [ 9917.930857] Lustre: lustre-MDT0001: new disk, initializing [ 9918.035353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9918.079123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9918.086535] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9923.619357] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 9935.254758] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9941.726517] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9949.033375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9949.379578] Lustre: lustre-OST0000: new disk, initializing [ 9949.384094] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9949.389636] Lustre: 344172:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9949.473166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9955.876161] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [ 9958.457452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9958.464944] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9958.613589] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9965.440129] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9970.451656] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9970.709627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9975.191776] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9975.446390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9977.318812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9977.326113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9977.327459] Lustre: Skipped 1 previous similar message [ 9982.437561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9982.453908] Lustre: Skipped 2 previous similar messages [ 9983.466514] Lustre: server umount lustre-OST0000 complete [ 9987.552677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9987.570185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9987.578512] Lustre: Skipped 1 previous similar message [ 9992.677114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9992.681373] Lustre: Skipped 1 previous similar message [ 9997.795821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9997.807154] Lustre: Skipped 1 previous similar message [10001.375204] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10001.577874] Lustre: server umount lustre-MDT0000 complete [10004.968945] LustreError: 342196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862507 with bad export cookie 2013086701957661854 [10004.973332] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10004.981403] LustreError: 342196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10005.232567] Lustre: server umount lustre-MDT0001 complete [10013.844689] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 16:28:34 (1787862514) [10023.267353] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 16:28:43 (1787862523) [10032.592166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10033.312829] LustreError: 346930:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10033.340231] LustreError: 346930:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [10033.442372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10038.756179] LustreError: 346931:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10038.873521] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [10043.871974] LustreError: 346930:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10048.650054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10048.970514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10053.996192] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [10062.601771] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10069.174518] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10076.911430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10077.299825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10084.209230] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [10092.256210] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10101.467910] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10104.120577] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10105.903843] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10108.009578] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10110.114856] Lustre: Failing over lustre-MDT0000 [10110.432919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10110.437247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10110.450268] Lustre: Skipped 1 previous similar message [10110.455407] LustreError: 346930:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10110.500267] Lustre: server umount lustre-MDT0000 complete [10114.532858] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10114.534885] LustreError: 347660:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10114.565474] LustreError: 347660:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [10124.772435] LustreError: 346926:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10124.810868] LustreError: 346926:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [10129.743238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10129.855146] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10130.170971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10130.233948] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10132.285961] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10134.388711] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [10135.536495] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10135.574454] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10135.629337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10145.080723] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10146.897939] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10150.989749] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10153.262094] Lustre: Failing over lustre-MDT0000 [10153.332845] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.32@tcp (stopping) [10153.562851] Lustre: server umount lustre-MDT0000 complete [10156.004454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10156.025355] Lustre: Skipped 3 previous similar messages [10156.037971] LustreError: 348808:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10156.047898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10156.071535] LustreError: 348808:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [10171.295945] Lustre: 340427:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787862658/real 1787862658] req@00000000113561ca x1874709566285312/t0(0) o400->MGC192.168.206.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787862674 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10171.337457] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10173.989295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10181.600737] LustreError: 340424:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000022e81d4 x1874709566290688/t0(0) o250->MGC192.168.206.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10181.968278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10182.056237] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10183.990389] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10186.737942] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [10187.255910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10187.269881] Lustre: Skipped 2 previous similar messages [10187.311575] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10187.380499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10197.416644] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10199.497886] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10207.729595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10207.734033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10207.736409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10207.747651] Lustre: Skipped 2 previous similar messages [10211.828508] Lustre: server umount lustre-MDT0000 complete [10212.837252] LustreError: 347660:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10212.869804] LustreError: 347660:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 23 previous similar messages [10215.579034] LustreError: 346910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862718 with bad export cookie 2013086701957664563 [10215.584890] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10215.595911] LustreError: 346910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10216.081336] Lustre: server umount lustre-MDT0001 complete [10230.301441] Lustre: server umount lustre-OST0000 complete [10248.399484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10249.026968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10254.099289] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [10263.492268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10268.788442] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [10277.002441] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10283.307827] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10291.343157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10293.979994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10298.816179] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [10305.733424] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10319.843583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10319.848725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10319.868152] LustreError: Skipped 1 previous similar message [10319.874272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10319.879099] Lustre: Skipped 2 previous similar messages [10321.894825] Lustre: server umount lustre-OST0000 complete [10325.475472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10331.685665] Lustre: server umount lustre-MDT0000 complete [10335.201487] LustreError: 353274:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10335.231759] LustreError: 353274:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [10335.877498] LustreError: 353259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862838 with bad export cookie 2013086701957665312 [10335.883046] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10335.889966] LustreError: 353259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10336.442426] Lustre: server umount lustre-MDT0001 complete [10348.189861] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [10351.066634] Key type lgssc unregistered [10351.314629] LNet: 356816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10351.330168] LNetError: 356816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10351.348280] LNet: Removed LNI 192.168.206.132@tcp [10352.564347] Key type .llcrypt unregistered [10352.568297] Key type ._llcrypt unregistered [10383.363476] Key type ._llcrypt registered [10383.370239] Key type .llcrypt registered [10383.503722] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [10384.841176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10384.912860] alg: No test for adler32 (adler32-zlib) [10386.103400] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [10386.385060] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [10388.137861] Key type lgssc registered [10389.694591] Lustre: Echo OBD driver; http://www.lustre.org/ [10399.012729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10399.047981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10400.702163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10405.570872] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [10410.978839] LustreError: 358333:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10411.008738] LustreError: 358333:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10415.329279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10415.732448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10420.901796] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [10429.341613] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10436.129376] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10443.303662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10443.630880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10445.675726] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10451.202349] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [10459.893582] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10471.394308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10471.409327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10471.427993] Lustre: Skipped 1 previous similar message [10472.245206] Lustre: server umount lustre-MDT0000 complete [10475.957705] LustreError: 358316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862978 with bad export cookie 11936941516539059993 [10475.960835] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10475.966740] LustreError: 358316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10476.327101] Lustre: server umount lustre-MDT0001 complete [10491.829664] Lustre: server umount lustre-OST0000 complete [10507.430725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10507.911842] LustreError: 362047:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10508.032384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10512.964891] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [10513.390024] LustreError: 362048:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10517.474767] LustreError: 362047:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10522.505081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10522.607117] LustreError: 362048:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10522.941058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10527.883205] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [10535.782250] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10541.588782] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10548.495574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10548.829991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10552.954803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10555.389563] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [10562.144566] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10573.798280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10573.800738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10573.810164] Lustre: Skipped 3 previous similar messages [10573.836450] Lustre: Skipped 4 previous similar messages [10578.913422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10578.923314] Lustre: Skipped 2 previous similar messages [10579.877634] Lustre: server umount lustre-MDT0000 complete [10583.171992] LustreError: 362027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863085 with bad export cookie 11936941516539061106 [10583.180511] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10583.183494] LustreError: 362027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10583.447789] Lustre: server umount lustre-MDT0001 complete [10596.525318] Lustre: server umount lustre-OST0000 complete [10612.654183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10613.360965] LustreError: 365958:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10613.462563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10618.990618] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [10629.061351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10634.286642] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [10643.521388] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10650.634682] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10658.784177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10659.139368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10659.145128] Lustre: Skipped 1 previous similar message [10660.159231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10665.786712] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [10674.008806] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10679.784313] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10679.796786] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10679.807970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10679.834772] Lustre: Skipped 1 previous similar message [10681.483530] Lustre: server umount lustre-OST0000 complete [10685.923081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10685.936439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10685.974495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10685.979715] Lustre: Skipped 1 previous similar message [10690.019880] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10691.399470] Lustre: server umount lustre-MDT0000 complete [10695.138199] LustreError: 365958:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10695.157168] LustreError: 365958:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [10695.378676] LustreError: 365939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863198 with bad export cookie 11936941516539062121 [10695.388871] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10695.406743] LustreError: 365939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10695.909929] Lustre: server umount lustre-MDT0001 complete [10708.099677] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [10710.942959] Key type lgssc unregistered [10711.238411] LNet: 369492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10711.244874] LNetError: 369492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10712.294889] LNet: Removed LNI 192.168.206.132@tcp [10713.295738] Key type .llcrypt unregistered [10713.303346] Key type ._llcrypt unregistered [10741.049959] Key type ._llcrypt registered [10741.054872] Key type .llcrypt registered [10741.252109] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [10742.327325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10742.415291] alg: No test for adler32 (adler32-zlib) [10743.682753] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [10743.983897] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [10745.719179] Key type lgssc registered [10746.884948] Lustre: Echo OBD driver; http://www.lustre.org/ [10756.209470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10756.222470] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10756.233117] LustreError: 370932:0:(osd_handler.c:8550:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10756.269560] LustreError: 370932:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10756.280659] LustreError: 370932:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10756.290682] LustreError: 370932:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10756.304639] LustreError: 370932:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10764.150219] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10764.161389] LustreError: 371330:0:(osd_handler.c:8550:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10764.180873] LustreError: 371330:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10764.186500] LustreError: 371330:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10764.196228] LustreError: 371330:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10764.206805] LustreError: 371330:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10788.678605] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [10792.051620] Key type lgssc unregistered [10792.392537] LNet: 372338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10792.402600] LNetError: 372338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10792.434651] LNet: Removed LNI 192.168.206.132@tcp [10793.435466] Key type .llcrypt unregistered [10793.443152] Key type ._llcrypt unregistered [10812.449468] Key type ._llcrypt registered [10812.452483] Key type .llcrypt registered [10812.545278] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_hostid [10828.329484] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [10829.786349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10829.797751] alg: No test for adler32 (adler32-zlib) [10830.859960] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [10831.147364] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [10832.871149] Key type lgssc registered [10834.324674] Lustre: Echo OBD driver; http://www.lustre.org/ [10867.768108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10867.813245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10869.099216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10869.125219] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10869.237896] Lustre: lustre-MDT0000: new disk, initializing [10869.325279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10869.345227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10873.486682] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [10883.243187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10883.477293] Lustre: 375494:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10883.549986] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10883.554164] Lustre: Skipped 1 previous similar message [10883.627957] Lustre: lustre-MDT0001: new disk, initializing [10883.770354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10883.848336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10883.868055] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10889.242420] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [10899.514435] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10906.118883] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10915.356565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10915.773927] Lustre: lustre-OST0000: new disk, initializing [10915.780276] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10915.787415] Lustre: 377463:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10915.972124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10921.491615] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [10924.627618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10924.645098] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10924.777594] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10931.678466] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10936.117947] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10936.377986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10940.758306] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10941.028925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10945.510424] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10945.525611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10945.528857] Lustre: Skipped 1 previous similar message [10945.552414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10950.645728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10950.660748] Lustre: Skipped 2 previous similar messages [10955.755031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10955.771417] Lustre: Skipped 1 previous similar message [10957.279307] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10957.653255] Lustre: server umount lustre-OST0000 complete [10965.985139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10965.997767] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10966.008546] Lustre: Skipped 1 previous similar message [10966.011283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10966.020202] Lustre: Skipped 1 previous similar message [10968.627619] Lustre: server umount lustre-MDT0000 complete [10972.116396] LustreError: 379004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863474 with bad export cookie 11261564473875338320 [10972.117446] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10972.123264] LustreError: 379004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10972.374961] Lustre: server umount lustre-MDT0001 complete [10981.130598] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10983.015216] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 16:44:43 (1787863483) [10989.819408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10990.189898] LustreError: 379922:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10990.224073] LustreError: 379922:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [10990.279085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10995.557725] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [10995.691839] LustreError: 379923:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10999.781552] LustreError: 379922:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11004.899117] LustreError: 379923:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11005.687709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11006.118186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11011.893256] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [11020.585138] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11027.477538] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11035.086769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11035.449444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11042.021655] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [11050.301564] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11057.673696] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11057.823195] Lustre: lustre-OST0001: new disk, initializing [11057.826534] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11057.832948] Lustre: 382763:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11057.910192] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11064.673908] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [11068.003986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11068.012930] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11068.117365] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11074.904157] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11083.092636] Lustre: lustre-OST0001: Not available for connect from 192.168.206.32@tcp (stopping) [11083.753388] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11083.756653] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11083.767861] LustreError: Skipped 1 previous similar message [11083.798853] Lustre: Skipped 1 previous similar message [11086.036641] Lustre: server umount lustre-OST0001 complete [11088.194796] LustreError: 383247:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11099.114717] LustreError: 383247:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11099.132134] LustreError: 383247:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [11104.231094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11104.241128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11104.259595] Lustre: Skipped 1 previous similar message [11104.276699] Lustre: Skipped 3 previous similar messages [11109.874574] Lustre: server umount lustre-OST0000 complete [11114.470352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11114.480736] Lustre: Skipped 1 previous similar message [11124.707897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11124.716867] Lustre: Skipped 7 previous similar messages [11128.287192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11128.565184] Lustre: server umount lustre-MDT0000 complete [11129.824677] LustreError: 379922:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11129.852776] LustreError: 379922:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [11132.259082] LustreError: 379903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863634 with bad export cookie 11261564473875339202 [11132.267059] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11132.272774] LustreError: 379903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11138.740245] Lustre: server umount lustre-MDT0001 complete [11150.411461] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [11153.262779] Key type lgssc unregistered [11153.660395] LNet: 384755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11153.673398] LNetError: 384755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11154.734653] LNet: Removed LNI 192.168.206.132@tcp [11155.867723] Key type .llcrypt unregistered [11155.873358] Key type ._llcrypt unregistered [11211.298717] Key type ._llcrypt registered [11211.302958] Key type .llcrypt registered [11211.449988] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [11212.862133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11212.983561] alg: No test for adler32 (adler32-zlib) [11213.990583] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [11214.274588] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [11216.015140] Key type lgssc registered [11217.275574] Lustre: Echo OBD driver; http://www.lustre.org/ [11226.124333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11226.143314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11227.361535] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11227.379612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11227.629253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11232.596567] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [11242.600903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11242.888689] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11242.959549] Lustre: 387555:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11243.439249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11248.578436] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [11256.772757] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11262.627494] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11264.992529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11265.015226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11265.025899] Lustre: Skipped 1 previous similar message [11270.112307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11270.121276] Lustre: Skipped 1 previous similar message [11270.735818] Lustre: server umount lustre-MDT0000 complete [11274.414725] LustreError: 387548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863777 with bad export cookie 7335795944039973224 [11274.427474] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11274.434640] LustreError: 387548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11274.807679] Lustre: server umount lustre-MDT0001 complete [11284.845413] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 16:49:45 (1787863785) [11292.030304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11292.923329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11297.629279] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [11298.274301] LustreError: 390068:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11298.286212] LustreError: 390068:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11303.396320] LustreError: 390067:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11306.020094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11306.468443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11310.786547] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [11318.032364] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11324.285930] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11332.280694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11332.554353] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11332.827346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11339.384161] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [11347.525970] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11356.081310] Lustre: Permanently deactivating lustre-OST0000 [11356.111723] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11361.914218] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11363.999157] LustreError: 393006:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11364.013229] LustreError: 393006:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11364.027084] LustreError: 393006:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11368.213710] LustreError: 393055:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11368.226407] LustreError: 393055:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11372.667982] Lustre: server umount lustre-OST0000 complete [11376.343296] LustreError: 393304:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11376.353654] LustreError: 393304:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11380.710523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11380.718612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11380.735858] Lustre: Skipped 2 previous similar messages [11380.773992] Lustre: Skipped 1 previous similar message [11385.824351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11385.835452] Lustre: Skipped 1 previous similar message [11388.997283] Lustre: server umount lustre-MDT0000 complete [11390.944033] LustreError: 391087:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11390.969295] LustreError: 391087:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11393.586772] LustreError: 392759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863896 with bad export cookie 7335795944039973805 [11393.590820] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11393.603564] LustreError: 392759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11404.520340] Lustre: server umount lustre-MDT0001 complete [11414.234308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11419.498339] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [11422.566735] LustreError: 394466:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11430.451509] Lustre: 394807:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.132@tcp [11433.307259] Lustre: server umount MGS complete [11448.142419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11448.988555] LustreError: 395531:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11449.024110] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11449.036086] Lustre: Skipped 1 previous similar message [11449.142221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11454.436993] LustreError: 395532:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11454.456596] LustreError: 395532:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11454.627444] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [11464.680204] LustreError: 395531:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11464.701722] LustreError: 395531:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11465.397169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11465.797426] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11465.896807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11470.339558] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [11478.659930] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11484.699861] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11486.939207] Lustre: Permanently reactivating lustre-OST0000 [11486.958880] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11486.968633] Lustre: Skipped 2 previous similar messages [11514.848287] LustreError: 397317:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11522.930466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11523.365319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11529.895627] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [11537.542041] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11545.057413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11545.063093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11545.082704] LustreError: Skipped 1 previous similar message [11545.107775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11545.124463] Lustre: Skipped 1 previous similar message [11545.615607] Lustre: server umount lustre-OST0000 complete [11550.176338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11550.198272] Lustre: Skipped 2 previous similar messages [11555.303725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11555.318955] Lustre: Skipped 3 previous similar messages [11556.538343] Lustre: server umount lustre-MDT0000 complete [11560.423878] LustreError: 396560:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11560.446390] LustreError: 396560:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11560.760553] LustreError: 395513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864063 with bad export cookie 7335795944039974855 [11560.763091] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11560.770991] LustreError: 395513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11561.159658] Lustre: server umount lustre-MDT0001 complete [11575.008210] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [11578.650708] Key type lgssc unregistered [11579.008230] LNet: 399903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11579.023432] LNetError: 399903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11579.056168] LNet: Removed LNI 192.168.206.132@tcp [11580.218191] Key type .llcrypt unregistered [11580.220966] Key type ._llcrypt unregistered [11601.595660] Key type ._llcrypt registered [11601.597855] Key type .llcrypt registered [11601.702370] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_hostid [11619.034543] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [11620.349539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11620.477151] alg: No test for adler32 (adler32-zlib) [11621.501967] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [11621.693909] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [11623.415179] Key type lgssc registered [11624.546670] Lustre: Echo OBD driver; http://www.lustre.org/ [11664.045620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11664.071604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11665.429238] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11665.455721] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11665.597863] Lustre: lustre-MDT0000: new disk, initializing [11665.661311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11665.682798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11670.242618] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [11681.365149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11681.476595] Lustre: 403062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11681.518137] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11681.521132] Lustre: Skipped 1 previous similar message [11681.575985] Lustre: lustre-MDT0001: new disk, initializing [11681.679237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11681.721894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11681.731163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11686.215439] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [11696.588441] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11702.881706] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11710.531968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11710.951867] Lustre: lustre-OST0000: new disk, initializing [11710.956213] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11710.962505] Lustre: 405030:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11711.088132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11716.695149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11716.712835] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11716.850812] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11718.541589] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [11728.840767] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11734.761742] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11735.048129] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11740.801821] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11741.030254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11747.808734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11747.818588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11747.821926] LustreError: Skipped 1 previous similar message [11747.826693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11747.859900] Lustre: Skipped 1 previous similar message [11749.481675] Lustre: server umount lustre-OST0000 complete [11753.452214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11753.463733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11753.491070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11753.505426] Lustre: Skipped 1 previous similar message [11758.049143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11758.050775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11758.082442] Lustre: Skipped 1 previous similar message [11759.675863] Lustre: server umount lustre-MDT0000 complete [11763.581041] LustreError: 403054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864266 with bad export cookie 9297203300184792740 [11763.599150] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11763.604678] LustreError: 403054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11763.944718] Lustre: server umount lustre-MDT0001 complete [11773.742384] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11775.606349] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 16:57:56 (1787864276) [11782.226404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11782.624316] LustreError: 407490:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11782.638923] LustreError: 407490:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [11782.689179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11787.054437] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [11787.744986] LustreError: 407491:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11792.864204] LustreError: 407490:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11794.864962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11795.221931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11799.350309] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [11807.104385] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11813.394652] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11822.023690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11822.363330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11829.094418] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [11838.028460] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11840.038549] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11842.611478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11848.160274] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11848.170290] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11848.188272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11850.216841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11853.767602] Lustre: server umount lustre-OST0000 complete [11860.449744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11860.452160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11860.465937] Lustre: Skipped 2 previous similar messages [11860.477035] Lustre: Skipped 3 previous similar messages [11864.670160] Lustre: server umount lustre-MDT0000 complete [11865.572559] LustreError: 407491:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11865.608685] LustreError: 407491:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11868.390822] LustreError: 407472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864371 with bad export cookie 9297203300184793622 [11868.393049] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11868.411795] LustreError: 407472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11868.866936] Lustre: server umount lustre-MDT0001 complete [11881.894781] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [11885.186382] Key type lgssc unregistered [11885.511675] LNet: 411220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11885.518449] LNetError: 411220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11885.542420] LNet: Removed LNI 192.168.206.132@tcp [11886.621538] Key type .llcrypt unregistered [11886.627713] Key type ._llcrypt unregistered [11911.639751] Key type ._llcrypt registered [11911.643300] Key type .llcrypt registered [11911.746794] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [11912.067391] Key type .llcrypt unregistered [11912.072867] Key type ._llcrypt unregistered [11928.640804] Key type ._llcrypt registered [11928.650388] Key type .llcrypt registered [11928.793289] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [11930.039405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11930.070023] alg: No test for adler32 (adler32-zlib) [11931.122661] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [11931.492836] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [11933.207191] Key type lgssc registered [11934.335340] Lustre: Echo OBD driver; http://www.lustre.org/ [11942.029197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11942.048829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11943.433311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11947.673620] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [11953.636644] LustreError: 413436:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11953.665071] LustreError: 413436:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11956.208526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11956.545780] LustreError: 413437:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11956.587138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11961.828930] LustreError: 413436:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11961.840094] LustreError: 413436:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11963.095418] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [11966.951055] LustreError: 414707:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11966.977270] LustreError: 414707:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11969.019862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11970.880646] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11974.120420] LustreError: 414707:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11974.133693] LustreError: 414707:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [11977.821164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11978.206588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11983.446397] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [11993.572785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11993.589944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11993.593421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11993.632231] Lustre: Skipped 1 previous similar message [11998.567994] Lustre: server umount lustre-OST0000 complete [12003.812112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12003.815212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12003.822437] Lustre: Skipped 1 previous similar message [12003.822818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12003.852218] Lustre: Skipped 2 previous similar messages [12008.873953] Lustre: server umount lustre-MDT0000 complete [12008.947588] LustreError: 413431:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12008.985304] LustreError: 413431:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12012.121421] LustreError: 413417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864514 with bad export cookie 5013544231973354991 [12012.129204] LustreError: 413417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12012.129296] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12012.417540] Lustre: server umount lustre-MDT0001 complete [12023.312754] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [12025.896211] Key type lgssc unregistered [12026.173567] LNet: 416599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12026.178325] LNetError: 416599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12026.199166] LNet: Removed LNI 192.168.206.132@tcp [12026.980594] Key type .llcrypt unregistered [12026.983180] Key type ._llcrypt unregistered [12049.400819] Key type ._llcrypt registered [12049.407326] Key type .llcrypt registered [12049.533769] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [12050.600663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12050.614524] alg: No test for adler32 (adler32-zlib) [12051.694257] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [12051.949809] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [12053.584538] Key type lgssc registered [12054.867560] Lustre: Echo OBD driver; http://www.lustre.org/ [12062.514932] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12062.537963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12078.175722] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12083.295353] LustreError: 417985:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.132@tcp: failed processing log, type 4: rc = -110 [12109.983241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12116.547866] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [12125.058732] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12132.348867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12132.806137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12132.887888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12137.137865] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [12137.954925] LustreError: 418010:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12137.972595] LustreError: 418010:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [12143.076284] LustreError: 418011:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12145.238779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12145.603867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12149.613722] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [12156.882177] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12162.814055] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12171.232332] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12171.247354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12171.266674] LustreError: 418001:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12171.366711] Lustre: server umount lustre-OST0000 complete [12176.359243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12176.371611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12176.380089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12177.377498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12177.381101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12182.497649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12182.506458] Lustre: Skipped 2 previous similar messages [12184.658581] Lustre: server umount lustre-MDT0000 complete [12186.592610] LustreError: 419028:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12186.608483] LustreError: 419028:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12188.190932] LustreError: 417991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864690 with bad export cookie 2096368893551193812 [12188.219767] LustreError: 417991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12188.224653] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12188.835918] Lustre: server umount lustre-MDT0001 complete [12202.224584] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [12205.174106] Key type lgssc unregistered [12205.435758] LNet: 421559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12205.444731] LNetError: 421559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12205.475369] LNet: Removed LNI 192.168.206.132@tcp [12206.372387] Key type .llcrypt unregistered [12206.373900] Key type ._llcrypt unregistered [12227.345403] Key type ._llcrypt registered [12227.346705] Key type .llcrypt registered [12227.447346] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [12228.393628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12228.511132] alg: No test for adler32 (adler32-zlib) [12229.521409] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [12229.718335] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [12231.367198] Key type lgssc registered [12232.497863] Lustre: Echo OBD driver; http://www.lustre.org/ [12239.069984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12239.081505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12240.459639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12244.273774] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [12250.593514] LustreError: 422955:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12250.611959] LustreError: 422955:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [12251.824051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12252.121496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12256.132619] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [12262.609629] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12267.144194] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12272.480289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12272.734994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12278.153170] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [12285.164275] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12286.062808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12291.555255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12291.557940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12291.569649] Lustre: Skipped 2 previous similar messages [12291.574839] Lustre: Skipped 1 previous similar message [12292.933390] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.32@tcp (stopping) [12292.942234] Lustre: Skipped 1 previous similar message [12295.592750] Lustre: server umount lustre-MDT0000 complete [12296.673016] LustreError: 422956:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12297.977457] LustreError: 422934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.32@tcp arrived at 1787864800 with bad export cookie 7349276737178818821 [12299.599398] LustreError: 422934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864802 with bad export cookie 7349276737178817925 [12299.608205] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12299.611053] LustreError: 422934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12301.792964] LustreError: 422952:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12301.793264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12301.819118] LustreError: 422952:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [12301.843735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12304.801393] LustreError: 422955:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12304.803735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12304.814072] LustreError: 422955:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12305.948706] Lustre: server umount lustre-OST0000 complete [12309.923859] LustreError: 422956:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12309.938379] LustreError: 422956:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12312.034789] Lustre: server umount lustre-MDT0001 complete [12320.954512] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [12323.255200] Key type lgssc unregistered [12323.503227] LNet: 426640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12323.513879] LNetError: 426640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12324.587604] LNet: Removed LNI 192.168.206.132@tcp [12325.432902] Key type .llcrypt unregistered [12325.440647] Key type ._llcrypt unregistered [12348.753743] Key type ._llcrypt registered [12348.756143] Key type .llcrypt registered [12348.849429] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [12349.707096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12349.828978] alg: No test for adler32 (adler32-zlib) [12350.915553] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [12351.149065] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [12352.823231] Key type lgssc registered [12353.907783] Lustre: Echo OBD driver; http://www.lustre.org/ [12361.532453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12361.560693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12362.895324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12366.966274] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [12372.456828] LustreError: 428058:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12372.475993] LustreError: 428058:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [12375.080803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12375.614296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12379.969675] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [12387.167836] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12392.708496] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12398.788514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12399.042612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12404.958918] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [12408.239849] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12410.710647] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12416.480103] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12416.486743] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12416.499246] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12418.377881] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.32@tcp (stopping) [12418.532735] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12421.247252] Lustre: server umount lustre-MDT0001 complete [12423.650194] LustreError: 428058:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12423.661667] LustreError: 428058:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12428.770816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12428.775210] LustreError: 429299:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12428.798629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12428.813231] Lustre: Skipped 2 previous similar messages [12432.133604] Lustre: server umount lustre-OST0000 complete [12433.905849] LustreError: 428058:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12435.570396] Lustre: server umount lustre-MDT0000 complete [12448.799882] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [12451.203787] Key type lgssc unregistered [12451.521786] LNet: 431745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12451.539443] LNetError: 431745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12451.566594] LNet: Removed LNI 192.168.206.132@tcp [12452.310432] Key type .llcrypt unregistered [12452.314183] Key type ._llcrypt unregistered [12478.315928] Key type ._llcrypt registered [12478.317210] Key type .llcrypt registered [12478.404449] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_hostid [12493.429537] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [12494.385936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12494.414970] alg: No test for adler32 (adler32-zlib) [12495.411607] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [12495.551918] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [12497.231384] Key type lgssc registered [12498.395279] Lustre: Echo OBD driver; http://www.lustre.org/ [12536.518559] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [12551.705682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12551.742516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12553.126515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12553.169415] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12553.220280] Lustre: lustre-MDT0000: new disk, initializing [12553.294573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12553.317944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12557.541804] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [12572.483786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12572.846589] Lustre: lustre-OST0000: new disk, initializing [12572.850403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12572.854440] Lustre: Skipped 1 previous similar message [12572.862799] Lustre: 436780:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12572.975056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12579.726449] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [12582.992616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12583.000610] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12583.030596] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12588.415258] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12602.813817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12602.887711] Lustre: 435761:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12602.960434] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12603.037827] Lustre: lustre-MDT0001: new disk, initializing [12603.170073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12603.207554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12603.223231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12607.244557] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [12615.881418] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12616.112675] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12620.776317] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12620.938736] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12626.243302] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.32@tcp (stopping) [12629.986224] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12629.989799] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12630.000302] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12630.006266] Lustre: Skipped 1 previous similar message [12631.350922] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.32@tcp (stopping) [12635.107980] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12635.117734] Lustre: Skipped 1 previous similar message [12639.199938] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12639.366409] Lustre: server umount lustre-MDT0001 complete [12645.348432] LustreError: 435772:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12645.364703] LustreError: 435772:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [12650.469820] LustreError: 435767:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12650.483216] LustreError: 435767:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12655.593979] LustreError: 435772:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12655.598185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12655.646556] Lustre: Skipped 1 previous similar message [12655.657713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12655.668265] Lustre: Skipped 1 previous similar message [12660.711535] LustreError: 435773:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12661.694760] Lustre: server umount lustre-OST0000 complete [12679.136579] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12679.537685] Lustre: server umount lustre-MDT0000 complete [12694.431490] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [12697.269162] Key type lgssc unregistered [12697.615857] LNet: 440535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12697.626177] LNetError: 440535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12697.646568] LNet: Removed LNI 192.168.206.132@tcp [12698.473445] Key type .llcrypt unregistered [12698.476055] Key type ._llcrypt unregistered [12736.560264] Key type ._llcrypt registered [12736.566627] Key type .llcrypt registered [12736.675683] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [12737.013292] Key type .llcrypt unregistered [12737.017138] Key type ._llcrypt unregistered [12752.298453] Key type ._llcrypt registered [12752.300503] Key type .llcrypt registered [12752.402278] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [12753.214165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12753.232524] alg: No test for adler32 (adler32-zlib) [12754.541695] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [12754.900725] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [12756.615212] Key type lgssc registered [12757.940569] Lustre: Echo OBD driver; http://www.lustre.org/ [12781.672106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12781.729499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12782.870723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12782.888947] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12782.942397] Lustre: lustre-MDT0000: new disk, initializing [12782.993126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12783.005974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12786.741742] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [12795.482296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12795.590294] Lustre: 443914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12795.661367] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12795.677560] Lustre: Skipped 1 previous similar message [12795.753641] Lustre: lustre-MDT0001: new disk, initializing [12795.848670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12795.879126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12795.888179] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12799.584734] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [12807.997591] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12812.838288] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12819.210227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12819.544595] Lustre: lustre-OST0000: new disk, initializing [12819.549136] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12819.561970] Lustre: 445880:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12819.691436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12825.484260] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [12828.740227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12828.751252] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12828.811928] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12834.471675] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12841.961763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12841.968894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12841.979147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12844.520423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12844.523459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12844.536638] Lustre: Skipped 1 previous similar message [12844.549386] Lustre: Skipped 2 previous similar messages [12846.291407] Lustre: server umount lustre-MDT0000 complete [12849.291884] LustreError: 443904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865352 with bad export cookie 15022458757884965807 [12849.304793] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12849.306735] LustreError: 443904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12849.552851] Lustre: server umount lustre-MDT0001 complete [12863.414271] Lustre: server umount lustre-OST0000 complete [12874.677359] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 17:16:15 (1787865375) [12882.262762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12882.722113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12886.704326] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [12888.036762] LustreError: 447908:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12888.060920] LustreError: 447908:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [12893.156137] LustreError: 447907:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12895.470412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12895.685344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12900.693804] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [12910.003195] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12915.909966] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12935.985473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12936.234895] Lustre: Found index 0 for lustre-OST0000, updating log [12936.417634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12942.468989] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [12949.466872] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12953.848709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12961.250137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12961.253756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12961.290495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12966.291180] Lustre: server umount lustre-OST0000 complete [12981.825605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12981.849699] Lustre: Skipped 1 previous similar message [12981.856896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12981.862877] Lustre: Skipped 4 previous similar messages [12986.849454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12986.853631] Lustre: Skipped 1 previous similar message [12988.189424] Lustre: server umount lustre-MDT0000 complete [12988.908886] LustreError: 450411:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12988.932358] LustreError: 450411:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12991.646884] LustreError: 447890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865494 with bad export cookie 15022458757884967200 [12991.647831] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12991.655895] LustreError: 447890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12992.089730] Lustre: server umount lustre-MDT0001 complete [13000.953832] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 17:18:21 (1787865501) [13004.767527] LNet: 1 local NIs in recovery (showing 1): 192.168.206.132@tcp [13625.066838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13625.586539] LustreError: 454318:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13625.602510] LustreError: 454318:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [13625.684474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13630.225570] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [13630.951761] LustreError: 454319:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13636.069251] LustreError: 454318:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13639.574893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13640.114698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13644.822484] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [13653.088670] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13659.340601] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13679.916805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13680.244226] Lustre: Found index 0 for lustre-OST0000, updating log [13680.594072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13683.682543] LNet: 1 local NIs in recovery (showing 1): 192.168.206.132@tcp [13687.703428] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [13696.463940] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13699.490858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13699.520576] Lustre: Skipped 1 previous similar message [13699.527200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13704.813919] Lustre: server umount lustre-OST0000 complete [13710.177243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13710.179850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13710.193067] Lustre: Skipped 1 previous similar message [13710.210147] Lustre: Skipped 1 previous similar message [13712.225637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13712.234471] Lustre: Skipped 3 previous similar messages [13715.602749] Lustre: server umount lustre-MDT0000 complete [13716.327769] LustreError: 455046:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13716.348054] LustreError: 455046:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [13719.605634] LustreError: 454298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866222 with bad export cookie 15022458757884968285 [13719.609461] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13719.625815] LustreError: 454298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13719.991816] Lustre: server umount lustre-MDT0001 complete [13744.101319] LNet: 1 local NIs in recovery (showing 1): 192.168.206.132@tcp [13751.579562] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_hostid [13762.801234] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [13805.414903] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 17:31:46 (1787866306) [13824.111548] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [13826.697434] Key type lgssc unregistered [13827.040217] LNet: 460939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13827.052081] LNetError: 460939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13939.749874] LNet: Removed LNI 192.168.206.132@tcp [13940.886939] Key type .llcrypt unregistered [13940.890371] Key type ._llcrypt unregistered [13959.150726] Key type ._llcrypt registered [13959.154173] Key type .llcrypt registered [13959.280964] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [13960.782645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13960.804303] alg: No test for adler32 (adler32-zlib) [13961.809588] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [13962.039970] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [13963.735159] Key type lgssc registered [13964.709731] Lustre: Echo OBD driver; http://www.lustre.org/ [13969.742930] LNet: Added LNI 192.168.206.132@tcp100 [8/256/0/180] [13970.622476] LNet: Added LNI 192.168.206.132@tcp101 [8/256/0/180] [13971.875532] LNet: Added LNI 192.168.206.132@tcp102 [8/256/0/180] [13972.886419] LNet: Added LNI 192.168.206.132@tcp103 [8/256/0/180] [13973.839728] LNet: Added LNI 192.168.206.132@tcp104 [8/256/0/180] [13975.135548] LNet: Added LNI 192.168.206.132@tcp105 [8/256/0/180] [13976.342077] LNet: Added LNI 192.168.206.132@tcp106 [8/256/0/180] [13977.104064] LNet: Added LNI 192.168.206.132@tcp107 [8/256/0/180] [13978.030513] LNet: Added LNI 192.168.206.132@tcp108 [8/256/0/180] [13978.887293] LNet: Added LNI 192.168.206.132@tcp109 [8/256/0/180] [13979.874046] LNet: Added LNI 192.168.206.132@tcp110 [8/256/0/180] [13980.856805] LNet: Added LNI 192.168.206.132@tcp111 [8/256/0/180] [13981.964376] LNet: Added LNI 192.168.206.132@tcp112 [8/256/0/180] [13983.068785] LNet: Added LNI 192.168.206.132@tcp113 [8/256/0/180] [13983.919904] LNet: Added LNI 192.168.206.132@tcp114 [8/256/0/180] [13985.111842] LNet: Added LNI 192.168.206.132@tcp115 [8/256/0/180] [13986.497439] LNet: Added LNI 192.168.206.132@tcp116 [8/256/0/180] [13987.574560] LNet: Added LNI 192.168.206.132@tcp117 [8/256/0/180] [13988.569646] LNet: Added LNI 192.168.206.132@tcp118 [8/256/0/180] [13989.762499] LNet: Added LNI 192.168.206.132@tcp119 [8/256/0/180] [13990.831903] LNet: Added LNI 192.168.206.132@tcp120 [8/256/0/180] [13992.112930] LNet: Added LNI 192.168.206.132@tcp121 [8/256/0/180] [13993.284219] LNet: Added LNI 192.168.206.132@tcp122 [8/256/0/180] [13994.344565] LNet: Added LNI 192.168.206.132@tcp123 [8/256/0/180] [13995.689372] LNet: Added LNI 192.168.206.132@tcp124 [8/256/0/180] [13996.889491] LNet: Added LNI 192.168.206.132@tcp125 [8/256/0/180] [13997.710948] LNet: Added LNI 192.168.206.132@tcp126 [8/256/0/180] [13998.757226] LNet: Added LNI 192.168.206.132@tcp127 [8/256/0/180] [14000.060118] LNet: Added LNI 192.168.206.132@tcp128 [8/256/0/180] [14001.134202] LNet: Added LNI 192.168.206.132@tcp129 [8/256/0/180] [14002.232411] LNet: Added LNI 192.168.206.132@tcp130 [8/256/0/180] [14003.317521] LNet: Added LNI 192.168.206.132@tcp131 [8/256/0/180] [14004.478659] LNet: Added LNI 192.168.206.132@tcp132 [8/256/0/180] [14005.339576] LNet: Added LNI 192.168.206.132@tcp133 [8/256/0/180] [14006.057458] LNet: Added LNI 192.168.206.132@tcp134 [8/256/0/180] [14006.921099] LNet: Added LNI 192.168.206.132@tcp135 [8/256/0/180] [14013.541893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14013.558530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14014.668249] Lustre: 463973:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.132@tcp: can't add NID 192.168.206.132@tcp130: rc = -75 [14014.752746] LNetError: 461429:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.132 [14014.774035] LNetError: Protocol error connecting to 192.168.206.132@tcp118 at host 192.168.206.132:988: is it running a compatible version of Lustre? [14014.798597] LNetError: Skipped 9 previous similar messages [14014.874675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14014.893353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14014.959901] Lustre: lustre-MDT0000: new disk, initializing [14015.060700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14015.083319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14015.327604] LNet: 8 local NIs in recovery (showing 8): 192.168.206.132@tcp120, 192.168.206.132@tcp122, 192.168.206.132@tcp123, 192.168.206.132@tcp124, 192.168.206.132@tcp127, 192.168.206.132@tcp128, 192.168.206.132@tcp126, 192.168.206.132@tcp118 [14015.328777] LNetError: 463997:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.132 [14015.354793] LNetError: 463997:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [14015.367342] LNetError: Protocol error connecting to 192.168.206.132@tcp125 at host 192.168.206.132:988: is it running a compatible version of Lustre? [14015.382058] LNetError: Skipped 4 previous similar messages [14017.376071] LNetError: 461428:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.132 [14017.387583] LNetError: 461428:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [14017.395419] LNetError: Protocol error connecting to 192.168.206.132@tcp126 at host 192.168.206.132:988: is it running a compatible version of Lustre? [14017.409092] LNetError: Skipped 1 previous similar message [14019.742670] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [14021.473360] LNetError: 463996:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.132 [14021.491231] LNetError: Protocol error connecting to 192.168.206.132@tcp126 at host 192.168.206.132:988: is it running a compatible version of Lustre? [14029.665649] LNetError: 461431:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.132 [14029.676117] LNetError: Protocol error connecting to 192.168.206.132@tcp126 at host 192.168.206.132:988: is it running a compatible version of Lustre? [14030.497775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14030.607774] Lustre: 463989:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14030.666642] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14030.673303] Lustre: Skipped 1 previous similar message [14030.735355] Lustre: lustre-MDT0001: new disk, initializing [14030.765877] Lustre: 464789:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.132@tcp: can't add NID 192.168.206.132@tcp130: rc = -75 [14030.772682] Lustre: 464789:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14030.867525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14030.905982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14030.917083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14034.153883] Lustre: 464956:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.132@tcp: can't add NID 192.168.206.132@tcp130: rc = -75 [14034.166988] Lustre: 464956:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14036.207447] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [14045.028892] LNetError: 463998:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.132 [14045.040816] LNetError: Protocol error connecting to 192.168.206.132@tcp126 at host 192.168.206.132:988: is it running a compatible version of Lustre? [14046.693664] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14052.994242] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14060.673522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14061.109694] Lustre: lustre-OST0000: new disk, initializing [14061.113392] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14061.121272] Lustre: 465965:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14061.219430] Lustre: 465970:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.132@tcp: can't add NID 192.168.206.132@tcp130: rc = -75 [14061.238469] Lustre: 465970:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14061.318274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14061.347599] LNetError: 461429:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.132 [14061.359453] LNetError: 461429:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [14061.401113] LNetError: Protocol error connecting to 192.168.206.132@tcp124 at host 192.168.206.132:988: is it running a compatible version of Lustre? [14061.422906] LNetError: Skipped 10 previous similar messages [14067.552745] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [14071.388211] Lustre: 466461:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.132@tcp: can't add NID 192.168.206.132@tcp130: rc = -75 [14071.407421] Lustre: 466461:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14071.536075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14071.553958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14071.905184] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14075.812510] LNet: 1 local NIs in recovery (showing 1): 192.168.206.132@tcp126 [14078.604972] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14092.258036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14092.262630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14092.264804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14092.277933] LustreError: Skipped 1 previous similar message [14093.767089] Lustre: server umount lustre-OST0000 complete [14095.139487] LNetError: 463996:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.132 [14095.167943] LNetError: 463996:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 18 previous similar messages [14095.186641] LNetError: Protocol error connecting to 192.168.206.132@tcp128 at host 192.168.206.132:988: is it running a compatible version of Lustre? [14095.202715] LNetError: Skipped 18 previous similar messages [14102.498124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14102.498851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14102.515667] Lustre: Skipped 1 previous similar message [14102.516615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14102.541722] Lustre: Skipped 2 previous similar messages [14104.380334] Lustre: server umount lustre-MDT0000 complete [14108.894148] LustreError: 463981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866611 with bad export cookie 17514843345864851051 [14108.915465] LustreError: 463981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14108.937313] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14109.363102] Lustre: server umount lustre-MDT0001 complete [14121.599660] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [14124.085879] Key type lgssc unregistered [14124.572187] LNet: 467879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14124.583832] LNetError: 467879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14124.595479] LNet: Removed LNI 192.168.206.132@tcp [14125.663143] LNet: Removed LNI 192.168.206.132@tcp100 [14125.668779] LNetError: 467879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14125.678424] LNetError: 467879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14125.684582] LNet: Removed LNI 192.168.206.132@tcp101 [14126.754179] LNet: Removed LNI 192.168.206.132@tcp102 [14126.757253] LNetError: 467879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14126.761140] LNetError: 467879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14126.767625] LNet: Removed LNI 192.168.206.132@tcp103 [14127.775679] LNet: Removed LNI 192.168.206.132@tcp104 [14128.799560] LNet: Removed LNI 192.168.206.132@tcp105 [14128.806394] LNetError: 467879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14128.813148] LNetError: 467879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14128.817892] LNet: Removed LNI 192.168.206.132@tcp106 [14129.823161] LNet: Removed LNI 192.168.206.132@tcp107 [14130.847955] LNet: Removed LNI 192.168.206.132@tcp108 [14131.872758] LNet: Removed LNI 192.168.206.132@tcp109 [14132.895853] LNet: Removed LNI 192.168.206.132@tcp110 [14132.900804] LNetError: 467879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14132.911973] LNetError: 467879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14132.922145] LNet: Removed LNI 192.168.206.132@tcp111 [14133.983132] LNet: Removed LNI 192.168.206.132@tcp112 [14135.007308] LNet: Removed LNI 192.168.206.132@tcp113 [14136.031300] LNet: Removed LNI 192.168.206.132@tcp114 [14137.055177] LNet: Removed LNI 192.168.206.132@tcp115 [14138.079180] LNet: Removed LNI 192.168.206.132@tcp116 [14139.103485] LNet: Removed LNI 192.168.206.132@tcp117 [14140.127150] LNet: Removed LNI 192.168.206.132@tcp118 [14141.151169] LNet: Removed LNI 192.168.206.132@tcp119 [14141.155102] LNetError: 467879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14141.165050] LNetError: 467879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14141.175223] LNet: Removed LNI 192.168.206.132@tcp120 [14142.241456] LNet: Removed LNI 192.168.206.132@tcp121 [14143.263153] LNet: Removed LNI 192.168.206.132@tcp122 [14144.287176] LNet: Removed LNI 192.168.206.132@tcp123 [14145.311217] LNet: Removed LNI 192.168.206.132@tcp124 [14146.335329] LNet: Removed LNI 192.168.206.132@tcp125 [14146.344118] LNet: Removed LNI 192.168.206.132@tcp126 [14147.359194] LNet: Removed LNI 192.168.206.132@tcp127 [14148.383163] LNet: Removed LNI 192.168.206.132@tcp128 [14149.407233] LNet: Removed LNI 192.168.206.132@tcp129 [14149.410459] LNet: Removed LNI 192.168.206.132@tcp130 [14149.419906] LNet: Removed LNI 192.168.206.132@tcp131 [14149.423361] LNet: Removed LNI 192.168.206.132@tcp132 [14149.426144] LNet: Removed LNI 192.168.206.132@tcp133 [14149.431082] LNet: Removed LNI 192.168.206.132@tcp134 [14149.454187] LNet: Removed LNI 192.168.206.132@tcp135 [14150.305343] Key type .llcrypt unregistered [14150.308817] Key type ._llcrypt unregistered [14174.294201] Key type ._llcrypt registered [14174.299509] Key type .llcrypt registered [14174.403552] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_hostid [14192.011986] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [14193.335682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14193.360385] alg: No test for adler32 (adler32-zlib) [14194.540156] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [14195.062205] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [14196.792422] Key type lgssc registered [14197.997826] Lustre: Echo OBD driver; http://www.lustre.org/ [14234.662833] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 17:38:55 (1787866735) [14255.028653] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [14257.893810] Key type lgssc unregistered [14258.235886] LNet: 471973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14258.240397] LNetError: 471973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14258.259692] LNet: Removed LNI 192.168.206.132@tcp [14259.223451] Key type .llcrypt unregistered [14259.225066] Key type ._llcrypt unregistered [14277.083528] Key type ._llcrypt registered [14277.087249] Key type .llcrypt registered [14277.232158] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [14278.498081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14278.520457] alg: No test for adler32 (adler32-zlib) [14279.533766] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [14279.716168] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [14281.367558] Key type lgssc registered [14282.528721] Lustre: Echo OBD driver; http://www.lustre.org/ [14292.735567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14292.803034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14294.130455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14294.154093] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14294.288370] Lustre: lustre-MDT0000: new disk, initializing [14294.410382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14294.446122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14299.054792] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [14309.920970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14310.082849] Lustre: 473188:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14310.156589] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14310.163369] Lustre: Skipped 1 previous similar message [14310.243304] Lustre: lustre-MDT0001: new disk, initializing [14310.358925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14310.397843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14310.422187] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14314.507987] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [14324.828510] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14330.837346] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14338.973031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14339.357849] Lustre: lustre-OST0000: new disk, initializing [14339.361578] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14339.366819] Lustre: 475157:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14339.473710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14346.341162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14346.358711] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14346.503067] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14347.675169] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [14357.234582] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14360.950774] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing lnet_if_list [14366.777917] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14366.815519] LNet: Added LNI 192.168.206.132@tcp10 [8/256/0/180] [14372.168286] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14372.194911] LNet: Added LNI 192.168.206.132@tcp11 [8/256/0/180] [14376.726932] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14376.750005] LNet: Added LNI 192.168.206.132@tcp12 [8/256/0/180] [14381.036980] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14381.060922] LNet: Added LNI 192.168.206.132@tcp13 [8/256/0/180] [14385.079223] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14385.111470] LNet: Added LNI 192.168.206.132@tcp14 [8/256/0/180] [14389.771536] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14389.794670] LNet: Added LNI 192.168.206.132@tcp15 [8/256/0/180] [14394.578368] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14394.599747] LNet: Added LNI 192.168.206.132@tcp16 [8/256/0/180] [14399.722988] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14399.754908] LNet: Added LNI 192.168.206.132@tcp17 [8/256/0/180] [14405.244655] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14405.271373] LNet: Added LNI 192.168.206.132@tcp18 [8/256/0/180] [14410.592603] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14410.623263] LNet: Added LNI 192.168.206.132@tcp19 [8/256/0/180] [14415.846400] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14415.878566] LNet: Added LNI 192.168.206.132@tcp20 [8/256/0/180] [14421.327994] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14421.353844] LNet: Added LNI 192.168.206.132@tcp21 [8/256/0/180] [14426.420834] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14426.466832] LNet: Added LNI 192.168.206.132@tcp22 [8/256/0/180] [14431.701772] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14431.739640] LNet: Added LNI 192.168.206.132@tcp23 [8/256/0/180] [14437.151708] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14437.187961] LNet: Added LNI 192.168.206.132@tcp24 [8/256/0/180] [14443.019336] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14443.058232] LNet: Added LNI 192.168.206.132@tcp25 [8/256/0/180] [14449.286388] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14449.310656] LNet: Added LNI 192.168.206.132@tcp26 [8/256/0/180] [14455.719255] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14455.740877] LNet: Added LNI 192.168.206.132@tcp27 [8/256/0/180] [14462.244245] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14462.272399] LNet: Added LNI 192.168.206.132@tcp28 [8/256/0/180] [14469.968424] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14470.009276] LNet: Added LNI 192.168.206.132@tcp29 [8/256/0/180] [14478.034580] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14478.088920] LNet: Added LNI 192.168.206.132@tcp30 [8/256/0/180] [14485.041397] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14485.089650] LNet: Added LNI 192.168.206.132@tcp31 [8/256/0/180] [14491.344779] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14491.384664] LNet: Added LNI 192.168.206.132@tcp32 [8/256/0/180] [14497.539440] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14497.569381] LNet: Added LNI 192.168.206.132@tcp33 [8/256/0/180] [14504.721816] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14504.761680] LNet: Added LNI 192.168.206.132@tcp34 [8/256/0/180] [14511.042440] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14511.065624] LNet: Added LNI 192.168.206.132@tcp35 [8/256/0/180] [14517.092394] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14517.135819] LNet: Added LNI 192.168.206.132@tcp36 [8/256/0/180] [14524.206433] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14524.252994] LNet: Added LNI 192.168.206.132@tcp37 [8/256/0/180] [14532.827853] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14532.891134] LNet: Added LNI 192.168.206.132@tcp38 [8/256/0/180] [14541.316400] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14541.360155] LNet: Added LNI 192.168.206.132@tcp39 [8/256/0/180] [14547.626836] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14547.657792] LNet: Added LNI 192.168.206.132@tcp40 [8/256/0/180] [14554.321841] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14554.348023] LNet: Added LNI 192.168.206.132@tcp41 [8/256/0/180] [14560.898823] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14560.926278] LNet: Added LNI 192.168.206.132@tcp42 [8/256/0/180] [14568.117172] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14568.155899] LNet: Added LNI 192.168.206.132@tcp43 [8/256/0/180] [14575.890486] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14575.913469] LNet: Added LNI 192.168.206.132@tcp44 [8/256/0/180] [14583.119813] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14583.158859] LNet: Added LNI 192.168.206.132@tcp45 [8/256/0/180] [14591.034531] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14591.058394] LNet: Added LNI 192.168.206.132@tcp46 [8/256/0/180] [14598.696172] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14598.736868] LNet: Added LNI 192.168.206.132@tcp47 [8/256/0/180] [14607.805653] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14607.853117] LNet: Added LNI 192.168.206.132@tcp48 [8/256/0/180] [14616.450831] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14616.478113] LNet: Added LNI 192.168.206.132@tcp49 [8/256/0/180] [14624.834958] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14624.872553] LNet: Added LNI 192.168.206.132@tcp50 [8/256/0/180] [14643.168844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14643.174461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14643.184548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14643.702117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14643.709671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14643.728034] Lustre: Skipped 1 previous similar message [14643.747441] Lustre: Skipped 1 previous similar message [14648.812420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14648.818769] Lustre: Skipped 2 previous similar messages [14649.469966] Lustre: server umount lustre-MDT0000 complete [14653.567256] LustreError: 473179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867156 with bad export cookie 6834156061417861977 [14653.583015] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14653.583381] LustreError: 473179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14653.922089] Lustre: server umount lustre-MDT0001 complete [14670.304595] Lustre: 472459:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787867156/real 1787867156] req@00000000d69fb533 x1874714198968704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787867172 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14670.323536] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14674.465147] Lustre: 472459:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787867161/real 1787867161] req@0000000084f448c8 x1874714198968832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787867177 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14675.136149] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [14679.521535] Lustre: 472462:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787867166/real 1787867166] req@000000004c9db904 x1874714198968960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787867182 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14683.852558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14684.641218] Lustre: 472461:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787867171/real 1787867172] req@000000000b1de57d x1874714198969216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787867187 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14699.377556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14703.922796] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [14704.418068] LustreError: 475151:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14704.457981] LustreError: 475151:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [14709.738971] LustreError: 475152:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14713.243783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14713.655273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14715.711941] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14718.663521] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [14723.952559] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14726.842934] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14731.975835] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14754.787212] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14754.790908] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14754.799104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14754.805437] Lustre: Skipped 1 previous similar message [14758.736070] Lustre: server umount lustre-OST0000 complete [14765.024531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14765.028891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14765.061373] Lustre: Skipped 1 previous similar message [14765.088586] Lustre: Skipped 2 previous similar messages [14775.276160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14775.283890] Lustre: Skipped 3 previous similar messages [14777.313527] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14777.938787] Lustre: server umount lustre-MDT0000 complete [14780.387320] LustreError: 499402:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14780.410373] LustreError: 499402:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [14781.589330] LustreError: 473984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867284 with bad export cookie 6834156061417864630 [14781.594103] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14781.599571] LustreError: 473984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14782.014780] Lustre: server umount lustre-MDT0001 complete [14794.359777] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [14797.482136] Key type lgssc unregistered [14798.009831] LNet: 502204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14798.026771] LNetError: 502204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14798.036299] LNet: Removed LNI 192.168.206.132@tcp [14798.042679] LNet: Removed LNI 192.168.206.132@tcp10 [14798.052367] LNet: Removed LNI 192.168.206.132@tcp11 [14798.062230] LNet: Removed LNI 192.168.206.132@tcp12 [14798.066339] LNet: Removed LNI 192.168.206.132@tcp13 [14798.072617] LNet: Removed LNI 192.168.206.132@tcp14 [14798.076212] LNet: Removed LNI 192.168.206.132@tcp15 [14798.078580] LNet: Removed LNI 192.168.206.132@tcp16 [14798.082655] LNet: Removed LNI 192.168.206.132@tcp17 [14798.086630] LNet: Removed LNI 192.168.206.132@tcp18 [14798.089406] LNet: Removed LNI 192.168.206.132@tcp19 [14798.095685] LNet: Removed LNI 192.168.206.132@tcp20 [14798.097632] LNet: Removed LNI 192.168.206.132@tcp21 [14798.101269] LNet: Removed LNI 192.168.206.132@tcp22 [14798.103903] LNet: Removed LNI 192.168.206.132@tcp23 [14798.107318] LNet: Removed LNI 192.168.206.132@tcp24 [14798.109814] LNet: Removed LNI 192.168.206.132@tcp25 [14798.112688] LNet: Removed LNI 192.168.206.132@tcp26 [14798.122518] LNet: Removed LNI 192.168.206.132@tcp27 [14798.131524] LNet: Removed LNI 192.168.206.132@tcp28 [14798.138182] LNet: Removed LNI 192.168.206.132@tcp29 [14798.142326] LNet: Removed LNI 192.168.206.132@tcp30 [14798.145813] LNet: Removed LNI 192.168.206.132@tcp31 [14798.150417] LNet: Removed LNI 192.168.206.132@tcp32 [14798.153593] LNet: Removed LNI 192.168.206.132@tcp33 [14798.156299] LNet: Removed LNI 192.168.206.132@tcp34 [14798.159415] LNet: Removed LNI 192.168.206.132@tcp35 [14798.162814] LNet: Removed LNI 192.168.206.132@tcp36 [14798.170757] LNet: Removed LNI 192.168.206.132@tcp37 [14798.173501] LNet: Removed LNI 192.168.206.132@tcp38 [14798.179378] LNet: Removed LNI 192.168.206.132@tcp39 [14798.182650] LNet: Removed LNI 192.168.206.132@tcp40 [14798.188200] LNet: Removed LNI 192.168.206.132@tcp41 [14798.191907] LNet: Removed LNI 192.168.206.132@tcp42 [14798.194749] LNet: Removed LNI 192.168.206.132@tcp43 [14798.200068] LNet: Removed LNI 192.168.206.132@tcp44 [14798.203444] LNet: Removed LNI 192.168.206.132@tcp45 [14798.205133] LNet: Removed LNI 192.168.206.132@tcp46 [14798.208251] LNet: Removed LNI 192.168.206.132@tcp47 [14798.211156] LNet: Removed LNI 192.168.206.132@tcp48 [14798.224211] LNet: Removed LNI 192.168.206.132@tcp49 [14798.234837] LNet: Removed LNI 192.168.206.132@tcp50 [14799.282166] Key type .llcrypt unregistered [14799.283470] Key type ._llcrypt unregistered [14820.879390] Key type ._llcrypt registered [14820.881395] Key type .llcrypt registered [14820.954791] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_hostid [14836.515272] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [14837.169679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14837.179199] alg: No test for adler32 (adler32-zlib) [14838.267870] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [14838.548406] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [14840.311168] Key type lgssc registered [14841.725789] Lustre: Echo OBD driver; http://www.lustre.org/ [14880.143263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14880.172993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14881.423532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14881.473397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14881.574974] Lustre: lustre-MDT0000: new disk, initializing [14881.627298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14881.648785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14886.132492] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [14896.627294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14896.776480] Lustre: 505403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14896.871290] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14896.881275] Lustre: Skipped 1 previous similar message [14897.018734] Lustre: lustre-MDT0001: new disk, initializing [14897.176960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14897.235838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14897.248168] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14902.018833] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [14911.741097] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14918.621620] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14927.067450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14927.294678] Lustre: *** cfs_fail_loc=90f, val=0*** [14927.365577] Lustre: lustre-OST0000: new disk, initializing [14927.368825] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14927.374781] Lustre: 507419:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14927.480112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14934.627727] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [14937.270744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14937.294648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14937.473138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14944.698683] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14958.071467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14958.082670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14958.086499] Lustre: Skipped 1 previous similar message [14958.098717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14958.113697] Lustre: Skipped 1 previous similar message [14959.873704] Lustre: server umount lustre-OST0000 complete [14968.304370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14968.320974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14968.327549] Lustre: Skipped 1 previous similar message [14968.347944] Lustre: Skipped 1 previous similar message [14970.525473] Lustre: server umount lustre-MDT0000 complete [14974.497541] LustreError: 505395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867477 with bad export cookie 7757929474448491093 [14974.503903] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14974.508703] LustreError: 505395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14974.850923] Lustre: server umount lustre-MDT0001 complete [14987.485679] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [14990.458763] Key type lgssc unregistered [14990.813399] LNet: 509326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14990.827279] LNetError: 509326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14990.845785] LNet: Removed LNI 192.168.206.132@tcp [14991.739234] Key type .llcrypt unregistered [14991.742378] Key type ._llcrypt unregistered [15013.741733] Key type ._llcrypt registered [15013.743099] Key type .llcrypt registered [15013.843563] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [15014.868938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15014.941059] alg: No test for adler32 (adler32-zlib) [15016.267669] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [15016.580672] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [15018.343189] Key type lgssc registered [15019.779816] Lustre: Echo OBD driver; http://www.lustre.org/ [15028.084773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15028.134766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15029.706228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15035.157821] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [15039.971706] LustreError: 510597:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15039.991833] LustreError: 510597:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [15045.093138] LustreError: 510598:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15045.712744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15046.115762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15050.841149] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [15060.626107] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15067.326265] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15074.178124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15074.491085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15080.613514] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [15087.403661] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15101.846803] loop0: detected capacity change from 0 to 204800000 [15102.046689] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15102.054984] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15102.080052] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15108.726945] loop0: detected capacity change from 0 to 204800000 [15108.953148] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15108.963947] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15108.971673] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15114.270785] loop0: detected capacity change from 0 to 204800000 [15120.061190] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15120.269838] Lustre: fs15246-OST0000: new disk, initializing [15120.276361] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15120.286130] Lustre: 514190:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15120.373824] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15128.065474] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [15138.926979] loop1: detected capacity change from 0 to 204800000 [15144.360889] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15144.487950] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15144.541695] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15144.610913] Lustre: fs15246-MDT0000: new disk, initializing [15144.711948] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15144.730167] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15144.804745] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15145.830557] Lustre: 515313:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000008d9fb1a7 x1874714971652480/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787867664 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15148.141088] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [15150.892558] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15150.903489] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15150.945192] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15176.167361] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15176.188114] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15178.533062] Lustre: server umount fs15246-MDT0000 complete [15181.287317] LustreError: 510594:0:(ldlm_lib.c:1200:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15186.425946] LustreError: 510592:0:(ldlm_lib.c:1200:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15188.317413] Lustre: server umount fs15246-OST0000 complete [15201.767583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15201.805066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15201.813589] Lustre: Skipped 1 previous similar message [15206.887892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15206.898078] Lustre: Skipped 1 previous similar message [15207.173862] Lustre: server umount lustre-OST0000 complete [15212.003694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15212.005413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15212.024576] Lustre: Skipped 2 previous similar messages [15212.036498] Lustre: Skipped 1 previous similar message [15217.128249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15217.137833] Lustre: Skipped 1 previous similar message [15217.899696] Lustre: server umount lustre-MDT0000 complete [15222.240641] LustreError: 513792:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15222.249605] LustreError: 513792:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [15222.276246] LustreError: 510580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867724 with bad export cookie 6586563286772772971 [15222.277712] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15222.288572] LustreError: 510580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15222.676583] Lustre: server umount lustre-MDT0001 complete [15235.911978] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [15238.946578] Key type lgssc unregistered [15239.221513] LNet: 518417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15239.232470] LNetError: 518417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15239.276995] LNet: Removed LNI 192.168.206.132@tcp [15240.228197] Key type .llcrypt unregistered [15240.231312] Key type ._llcrypt unregistered [15266.381725] Key type ._llcrypt registered [15266.383273] Key type .llcrypt registered [15266.442583] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [15267.019413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15267.055076] alg: No test for adler32 (adler32-zlib) [15268.046104] Lustre: Lustre: Build Version: 2.17.57_85_gfc6b239 [15268.185975] LNet: Added LNI 192.168.206.132@tcp [8/256/0/180] [15269.903700] Key type lgssc registered [15271.121940] Lustre: Echo OBD driver; http://www.lustre.org/ [15313.963338] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_hostid [15324.625500] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [15361.673993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15361.695926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15362.964831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15363.032414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15363.134966] Lustre: lustre-MDT0000: new disk, initializing [15363.266706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15363.301092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15367.405600] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [15381.533640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15381.824327] Lustre: 522887:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15381.974494] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15381.986039] Lustre: Skipped 1 previous similar message [15382.123691] Lustre: lustre-MDT0001: new disk, initializing [15382.346335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15382.416686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15382.436242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15389.632145] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [15401.894530] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15409.374149] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15417.553295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15417.879721] Lustre: lustre-OST0000: new disk, initializing [15417.885914] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15417.891399] Lustre: 524853:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15418.019332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15425.268832] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_default_debug -1 all [15427.642563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15427.649215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15427.721733] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15435.982671] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15441.423736] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15441.686217] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15447.387355] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15447.591772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15453.669097] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15453.670467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15453.721402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15453.733925] Lustre: Skipped 1 previous similar message [15456.115725] Lustre: server umount lustre-OST0000 complete [15463.905480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15463.915238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15463.938957] Lustre: Skipped 2 previous similar messages [15463.950680] Lustre: Skipped 1 previous similar message [15469.024319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15469.034543] Lustre: Skipped 1 previous similar message [15474.143124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15474.144466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15474.167261] Lustre: Skipped 1 previous similar message [15474.506034] Lustre: server umount lustre-MDT0000 complete [15478.496956] LustreError: 522879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867981 with bad export cookie 11368504987072699619 [15478.500531] LustreError: MGC192.168.206.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15478.517664] LustreError: 522879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15478.836920] Lustre: server umount lustre-MDT0001 complete [15505.583594] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing set_hostid [15515.845062] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing load_modules_local [15550.340610] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15280 sec ========= 18:00:50 (1787868050) [15552.385082] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:00:52 (1787868052) === [15557.575294] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:00:57 (1787868057) === [15603.984241] Lustre: DEBUG MARKER: oleg632-server.virtnet: executing unload_modules_local [15607.955507] Key type lgssc unregistered [15608.435325] LNet: 531516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15608.442924] LNetError: 531516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15608.459746] LNet: Removed LNI 192.168.206.132@tcp [15609.779504] Key type .llcrypt unregistered [15609.785649] Key type ._llcrypt unregistered