[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 565367215 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2544MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002308] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009130] LSM: Security Framework initializing [ 0.010070] Yama: becoming mindful. [ 0.011039] SELinux: Initializing. [ 0.012073] *** VALIDATE selinux *** [ 0.020301] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025553] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028113] *** VALIDATE tmpfs *** [ 0.029428] *** VALIDATE proc *** [ 0.030248] *** VALIDATE cgroup *** [ 0.031013] *** VALIDATE cgroup2 *** [ 0.032286] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033162] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036036] Spectre V2 : User space: Vulnerable [ 0.037013] Speculative Store Bypass: Vulnerable [ 0.040271] debug: unmapping init [mem 0xffffffffb8259000-0xffffffffb8260fff] [ 0.042172] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043619] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044027] ... version: 2 [ 0.045014] ... bit width: 48 [ 0.046015] ... generic registers: 4 [ 0.047014] ... value mask: 0000ffffffffffff [ 0.048016] ... max period: 00007fffffffffff [ 0.049019] ... fixed-purpose events: 3 [ 0.050014] ... event mask: 000000070000000f [ 0.051302] rcu: Hierarchical SRCU implementation. [ 0.053291] smp: Bringing up secondary CPUs ... [ 0.054557] x86: Booting SMP configuration: [ 0.055032] .... node #0, CPUs: #1 #2 #3 [ 0.060023] smp: Brought up 1 node, 4 CPUs [ 0.062026] smpboot: Max logical packages: 1 [ 0.063015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.155783] node 0 deferred pages initialised in 89ms [ 0.160027] devtmpfs: initialized [ 0.161272] x86/mm: Memory block size: 128MB [ 0.164065] gcov: version magic: 0x41383552 [ 0.166515] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.170095] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.173488] pinctrl core: initialized pinctrl subsystem [ 0.175231] [ 0.175989] ************************************************************* [ 0.178015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.181019] ** ** [ 0.183015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.185019] ** ** [ 0.188021] ** This means that this kernel is built to expose internal ** [ 0.190013] ** IOMMU data structures, which may compromise security on ** [ 0.192014] ** your system. ** [ 0.195018] ** ** [ 0.197015] ** If you see this message and you are not debugging the ** [ 0.200019] ** kernel, report this immediately to your vendor! ** [ 0.202017] ** ** [ 0.205020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.208019] ************************************************************* [ 0.211938] NET: Registered protocol family 16 [ 0.214508] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.217167] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.220099] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.223132] cpuidle: using governor menu [ 0.224766] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.228890] PCI: Using configuration type 1 for base access [ 0.230160] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.240217] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.242046] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.249020] cryptd: max_cpu_qlen set to 1000 [ 0.252200] ACPI: Added _OSI(Module Device) [ 0.254030] ACPI: Added _OSI(Processor Device) [ 0.255023] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.257016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.262933] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.270522] ACPI: Interpreter enabled [ 0.272080] ACPI: PM: (supports S0 S3 S4 S5) [ 0.274015] ACPI: Using IOAPIC for interrupt routing [ 0.276191] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.279430] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.289696] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.291045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.294044] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.297116] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.302435] acpiphp: Slot [2] registered [ 0.304147] acpiphp: Slot [5] registered [ 0.305155] acpiphp: Slot [6] registered [ 0.307197] acpiphp: Slot [7] registered [ 0.308125] acpiphp: Slot [8] registered [ 0.311222] acpiphp: Slot [9] registered [ 0.312122] acpiphp: Slot [10] registered [ 0.314168] acpiphp: Slot [3] registered [ 0.315136] acpiphp: Slot [4] registered [ 0.317120] acpiphp: Slot [11] registered [ 0.318181] acpiphp: Slot [12] registered [ 0.320125] acpiphp: Slot [13] registered [ 0.322143] acpiphp: Slot [14] registered [ 0.323200] acpiphp: Slot [15] registered [ 0.325148] acpiphp: Slot [16] registered [ 0.326153] acpiphp: Slot [17] registered [ 0.330168] acpiphp: Slot [18] registered [ 0.331118] acpiphp: Slot [19] registered [ 0.332127] acpiphp: Slot [20] registered [ 0.334132] acpiphp: Slot [21] registered [ 0.335159] acpiphp: Slot [22] registered [ 0.337179] acpiphp: Slot [23] registered [ 0.339144] acpiphp: Slot [24] registered [ 0.340133] acpiphp: Slot [25] registered [ 0.342137] acpiphp: Slot [26] registered [ 0.343140] acpiphp: Slot [27] registered [ 0.345142] acpiphp: Slot [28] registered [ 0.346173] acpiphp: Slot [29] registered [ 0.348145] acpiphp: Slot [30] registered [ 0.349111] acpiphp: Slot [31] registered [ 0.351070] PCI host bridge to bus 0000:00 [ 0.353024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.355032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.358024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.360027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.363040] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.365045] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.367190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.369074] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.372468] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.382907] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.388111] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.391028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.393027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.396039] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.399602] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.403820] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.405048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.410390] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.415020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.430017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.435018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.442056] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.451026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.459027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.481019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.492015] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.502025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.513026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.547026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.558784] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.569019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.578021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.594025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.607902] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.618023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.627022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.648021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.665472] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.675023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.682021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.699020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.711510] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.722022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.731280] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.751028] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.765522] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.768405] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.771375] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.774468] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.777263] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.782135] iommu: Default domain type: Passthrough [ 0.784706] SCSI subsystem initialized [ 0.787231] ACPI: bus type USB registered [ 0.788159] usbcore: registered new interface driver usbfs [ 0.791136] usbcore: registered new interface driver hub [ 0.793145] usbcore: registered new device driver usb [ 0.795205] pps_core: LinuxPPS API ver. 1 registered [ 0.797018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.800068] PTP clock support registered [ 0.803040] EDAC MC: Ver: 3.0.0 [ 0.805257] PCI: Using ACPI for IRQ routing [ 0.807888] NetLabel: Initializing [ 0.809016] NetLabel: domain hash size = 128 [ 0.811028] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.813105] NetLabel: unlabeled traffic allowed by default [ 0.815175] vgaarb: loaded [ 0.818239] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.819018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.827706] clocksource: Switched to clocksource kvm-clock [ 0.961223] VFS: Disk quotas dquot_6.6.0 [ 0.963163] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.966048] *** VALIDATE ramfs *** [ 0.967418] *** VALIDATE hugetlbfs *** [ 0.969500] pnp: PnP ACPI init [ 0.973629] pnp: PnP ACPI: found 6 devices [ 0.994326] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.999316] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.002600] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.005452] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.010028] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.012697] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.016621] NET: Registered protocol family 2 [ 1.019721] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.027309] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.032549] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.040509] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.045390] TCP: Hash tables configured (established 65536 bind 65536) [ 1.049985] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.054450] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.058348] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.062364] NET: Registered protocol family 1 [ 1.066071] RPC: Registered named UNIX socket transport module. [ 1.068746] RPC: Registered udp transport module. [ 1.071187] RPC: Registered tcp transport module. [ 1.073135] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.076098] NET: Registered protocol family 44 [ 1.078340] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.081392] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.084242] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.089291] PCI: CLS 0 bytes, default 64 [ 1.091739] Unpacking initramfs... [ 2.578608] debug: unmapping init [mem 0xffff9a197cc54000-0xffff9a197ffbffff] [ 2.583637] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.585256] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.587808] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.080703] Initialise system trusted keyrings [ 3.083273] Key type blacklist registered [ 3.085243] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.101846] zbud: loaded [ 3.107361] *** VALIDATE nfs *** [ 3.108727] *** VALIDATE nfs4 *** [ 3.110748] pstore: using deflate compression [ 3.116229] Platform Keyring initialized [ 3.247228] NET: Registered protocol family 38 [ 3.249298] Key type asymmetric registered [ 3.250381] Asymmetric key parser 'x509' registered [ 3.251588] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.255379] io scheduler mq-deadline registered [ 3.256583] io scheduler kyber registered [ 3.257869] io scheduler bfq registered [ 3.259498] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.261934] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.264342] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.266512] ACPI: Power Button [PWRF] [ 3.272759] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.279966] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.297852] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.310264] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.324057] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.351890] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.383363] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.388970] Non-volatile memory driver v1.3 [ 3.390663] Linux agpgart interface v0.103 [ 3.426779] virtio_blk virtio1: [vda] 146552 512-byte logical blocks (75.0 MB/71.6 MiB) [ 3.432050] vda: detected capacity change from 0 to 75034624 [ 3.450971] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.453726] vdb: detected capacity change from 0 to 1073741824 [ 3.470878] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.474278] vdc: detected capacity change from 0 to 2621440000 [ 3.487294] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.489082] vdd: detected capacity change from 0 to 2621440000 [ 3.504381] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.506310] vde: detected capacity change from 0 to 4294967296 [ 3.521382] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.523850] vdf: detected capacity change from 0 to 4294967296 [ 3.532846] libphy: Fixed MDIO Bus: probed [ 3.538394] usbcore: registered new interface driver usbserial_generic [ 3.540913] usbserial: USB Serial support registered for generic [ 3.542864] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.546393] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.547586] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.548914] mousedev: PS/2 mouse device common for all mice [ 3.550899] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.553405] rtc_cmos 00:05: RTC can wake from S4 [ 3.555276] rtc_cmos 00:05: registered as rtc0 [ 3.559029] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.561186] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.562132] intel_pstate: CPU model not supported [ 3.565593] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.569240] hid: raw HID events driver (C) Jiri Kosina [ 3.570596] usbcore: registered new interface driver usbhid [ 3.572302] usbhid: USB HID core driver [ 3.573390] drop_monitor: Initializing network drop monitor service [ 3.574617] Initializing XFRM netlink socket [ 3.575822] NET: Registered protocol family 10 [ 3.577740] Segment Routing with IPv6 [ 3.579466] NET: Registered protocol family 17 [ 3.581095] mpls_gso: MPLS GSO support [ 3.589253] RAS: Correctable Errors collector initialized. [ 3.591428] AVX version of gcm_enc/dec engaged. [ 3.592948] AES CTR mode by8 optimization enabled [ 3.668756] sched_clock: Marking stable (3668733047, 0)->(4664690576, -995957529) [ 3.675209] registered taskstats version 1 [ 3.677253] Loading compiled-in X.509 certificates [ 3.679241] zswap: loaded using pool lzo/zbud [ 3.706118] Key type big_key registered [ 3.720139] Key type encrypted registered [ 3.721464] ima: No TPM chip found, activating TPM-bypass! [ 3.723121] ima: Allocated hash algorithm: sha1 [ 3.724768] ima: No architecture policies found [ 3.726180] evm: Initialising EVM extended attributes: [ 3.727688] evm: security.selinux [ 3.728690] evm: security.ima [ 3.729692] evm: security.capability [ 3.730782] evm: HMAC attrs: 0x1 [ 3.733416] rtc_cmos 00:05: setting system clock to 2026-08-27 01:09:04 UTC (1787792944) [ 3.740061] debug: unmapping init [mem 0xffffffffb9203000-0xffffffffb93fffff] [ 3.745290] debug: unmapping init [mem 0xffffffffb7f82000-0xffffffffb8258fff] [ 3.754073] Write protecting the kernel read-only data: 28672k [ 3.757421] debug: unmapping init [mem 0xffffffffb6603000-0xffffffffb67fffff] [ 3.760294] debug: unmapping init [mem 0xffffffffb6f14000-0xffffffffb6ffffff] [ 3.797689] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.805964] systemd[1]: Detected virtualization kvm. [ 3.808350] systemd[1]: Detected architecture x86-64. [ 3.810325] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.837365] systemd[1]: No hostname configured. [ 3.839076] systemd[1]: Set hostname to . [ 3.841350] random: systemd: uninitialized urandom read (16 bytes read) [ 3.843562] systemd[1]: Initializing machine ID from random generator. [ 3.974950] random: systemd: uninitialized urandom read (16 bytes read) [ 3.978128] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.984193] random: systemd: uninitialized urandom read (16 bytes read) [ 3.986894] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.991696] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.717102] device-mapper: uevent: version 1.0.3 [ 4.719541] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.482908] virtio_net virtio0 ens2: renamed from eth0 [ 5.487129] random: fast init done [ 5.555187] scsi host0: ata_piix [ 5.574825] scsi host1: ata_piix [ 5.576943] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.579744] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.190769] dracut-initqueue[584]: RTNETLINK answers: File exists [ 10.236051] random: crng init done [ 10.237446] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.683738] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.901477] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.193104] SELinux: Disabled at runtime. [ 12.253653] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.261724] systemd[1]: Detected virtualization kvm. [ 12.263211] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.791159] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.794072] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.798343] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.802382] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.805069] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.815495] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.819198] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ 12.949745] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.289392] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.604165] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.615606] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.789491] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.804893] EDAC sbridge: Ver: 1.1.2 [ 15.549480] Key type dns_resolver registered [ 15.865938] NFS: Registering the id_resolver key type [ 15.867977] Key type id_resolver registered [ 15.869617] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg436-server login: [ 58.933537] libcfs: loading out-of-tree module taints kernel. [ 58.973479] Key type ._llcrypt registered [ 58.975590] Key type .llcrypt registered [ 59.064618] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 79.509599] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 81.205853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 81.235897] alg: No test for adler32 (adler32-zlib) [ 82.828799] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 83.646674] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 85.449502] Key type lgssc registered [ 87.655156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 107.824908] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 152.072988] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 169.476127] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 169.532047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 170.815256] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 170.840961] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 170.956196] Lustre: lustre-MDT0000: new disk, initializing [ 171.060179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 171.091859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 175.926445] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.512581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 190.602656] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 190.641253] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 190.646779] Lustre: Skipped 1 previous similar message [ 190.725929] Lustre: lustre-MDT0001: new disk, initializing [ 190.816814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 190.865589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 190.871409] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 197.039310] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.318181] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 215.751403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 216.304169] Lustre: lustre-OST0000: new disk, initializing [ 216.317321] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 216.326311] Lustre: 8449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 216.429090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 224.865316] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.403124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 226.425590] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 226.554924] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 233.409309] hrtimer: interrupt took 7298374 ns [ 242.449029] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 242.608111] Lustre: lustre-OST0001: new disk, initializing [ 242.612439] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 242.616967] Lustre: 9525:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 242.687287] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 250.162207] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 252.496062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 252.506697] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 252.604786] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 263.812293] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 271.543080] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 279.906752] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing check_logdir /tmp/testlogs/ [ 285.873982] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing yml_node [ 291.754359] Lustre: DEBUG MARKER: Client: 2.17.57.48 [ 294.736584] Lustre: DEBUG MARKER: MDS: 2.17.57.48 [ 298.235260] Lustre: DEBUG MARKER: OSS: 2.17.57.48 [ 300.144641] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 26 21:13:58 EDT 2026 [ 319.920071] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 321.552982] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 339.431214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 339.444890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.461194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 339.947482] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.978108] Lustre: Skipped 2 previous similar messages [ 339.985240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 339.989220] Lustre: Skipped 2 previous similar messages [ 344.604868] Lustre: server umount lustre-MDT0000 complete [ 348.975558] LustreError: 7460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793289 with bad export cookie 161300730911521228 [ 348.983389] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 348.988980] LustreError: 7460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 350.177608] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 350.178613] LustreError: 6518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 350.178761] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 350.200968] Lustre: Skipped 1 previous similar message [ 350.227103] LustreError: 6518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 353.188637] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 353.190174] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 353.209990] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 353.228196] Lustre: Skipped 2 previous similar messages [ 355.743775] Lustre: server umount lustre-MDT0001 complete [ 360.503220] Lustre: server umount lustre-OST0000 complete [ 365.638707] Lustre: server umount lustre-OST0001 complete [ 373.075839] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 383.999730] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 425.604123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 425.916522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 425.969559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 426.067121] Lustre: lustre-MDT0000: new disk, initializing [ 426.143185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 426.168393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 430.733329] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 442.744031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 442.899297] Lustre: 15080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 442.999520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 443.005993] Lustre: Skipped 1 previous similar message [ 443.127261] Lustre: lustre-MDT0001: new disk, initializing [ 443.235253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 443.268535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 443.289652] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 447.685688] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 458.282745] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 464.881744] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 472.135177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 472.489515] Lustre: lustre-OST0000: new disk, initializing [ 472.494458] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 472.508857] Lustre: 17048:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 472.667769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 474.170877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 474.185525] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 474.332761] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 478.763470] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 488.045356] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 492.285396] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 492.503349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 497.835949] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 498.049291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 500.195370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 500.200546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.209378] LustreError: Skipped 1 previous similar message [ 500.233299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 500.239479] Lustre: Skipped 1 previous similar message [ 510.434866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 510.452478] Lustre: Skipped 3 previous similar messages [ 514.016400] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 514.256649] Lustre: server umount lustre-OST0000 complete [ 520.161295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 520.166745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 520.175117] Lustre: Skipped 1 previous similar message [ 524.145381] Lustre: server umount lustre-MDT0000 complete [ 525.799875] LustreError: 15091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.834293] LustreError: 15091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 528.743403] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793469 with bad export cookie 161300730911522761 [ 528.744784] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 528.753602] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 529.210293] Lustre: server umount lustre-MDT0001 complete [ 535.151266] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 21:17:52 (1787793472) [ 538.191932] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 540.594850] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 21:17:58 (1787793478) [ 546.453121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 546.796337] LustreError: 19458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.846360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 550.544725] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 551.907139] LustreError: 19459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.816387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.722095] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 573.128744] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 578.863574] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 586.175181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 593.095659] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 602.575314] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 614.387844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 614.394465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 614.395225] Lustre: Skipped 2 previous similar messages [ 614.395825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 614.395832] Lustre: Skipped 2 previous similar messages [ 619.701712] Lustre: server umount lustre-OST0000 complete [ 630.212870] Lustre: server umount lustre-MDT0000 complete [ 634.534193] LustreError: 19441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793575 with bad export cookie 161300730911523636 [ 634.536536] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 634.554545] LustreError: 19441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 634.952053] Lustre: server umount lustre-MDT0001 complete [ 648.618894] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 653.149385] Key type lgssc unregistered [ 653.784984] LNet: 22994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.796775] LNetError: 22994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 653.811976] LNet: Removed LNI 192.168.204.136@tcp [ 655.310325] Key type .llcrypt unregistered [ 655.317150] Key type ._llcrypt unregistered [ 680.940191] Key type ._llcrypt registered [ 680.942653] Key type .llcrypt registered [ 681.063298] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 682.110705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 682.141504] alg: No test for adler32 (adler32-zlib) [ 683.168894] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 683.359601] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 684.984499] Key type lgssc registered [ 685.945248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 695.592489] loop: module loaded [ 695.705126] loop0: detected capacity change from 0 to 204800000 [ 695.943982] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 695.963242] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 696.366590] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 702.826490] loop0: detected capacity change from 0 to 204800000 [ 703.029083] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 703.044113] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 703.052518] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 709.116979] loop0: detected capacity change from 0 to 204800000 [ 715.722901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 715.761883] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 717.108135] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 717.206834] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 717.324055] Lustre: lustre-MDT0000: new disk, initializing [ 717.433133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 717.471154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 722.150652] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 731.242978] loop1: detected capacity change from 0 to 204800000 [ 737.573817] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 737.881211] Lustre: lustre-OST07c6: new disk, initializing [ 737.884503] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 737.894600] Lustre: Skipped 1 previous similar message [ 737.899562] Lustre: 26183:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 738.027303] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 743.588032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 743.597325] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 743.631233] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 744.472755] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 750.585090] Lustre: 25168:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 750.603624] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 750.613994] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 750.633420] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 750.654134] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 750.662575] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 767.226891] Lustre: 25170:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 767.239732] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 767.247745] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 767.255710] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 767.261453] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 767.270497] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 772.211750] Lustre: 25168:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 772.215838] Lustre: 25168:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 772.218592] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 772.222237] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.225685] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 772.230598] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.233808] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 772.237628] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.242076] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 772.245703] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.248906] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 772.251859] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.404055] Lustre: 26472:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 775.411169] Lustre: 26472:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 775.416989] Lustre: 26472:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 775.423297] Lustre: 26472:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.428680] Lustre: 26472:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 775.435238] Lustre: 26472:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.440303] Lustre: 26472:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 775.447190] Lustre: 26472:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.456402] Lustre: 26472:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 775.461465] Lustre: 26472:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.468077] Lustre: 26472:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 775.484046] Lustre: 26472:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.701396] Lustre: 26472:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 780.721124] Lustre: 26472:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 780.731279] Lustre: 26472:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 780.743791] Lustre: 26472:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.750905] Lustre: 26472:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 780.763419] Lustre: 26472:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.777070] Lustre: 26472:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 780.791426] Lustre: 26472:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.798171] Lustre: 26472:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 780.812458] Lustre: 26472:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.818413] Lustre: 26472:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 780.828834] Lustre: 26472:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.178451] Lustre: 26472:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 789.188926] Lustre: 26472:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 789.198728] Lustre: 26472:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 789.205375] Lustre: 26472:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 789.222382] Lustre: 26472:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 789.238038] Lustre: 26472:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 789.246436] Lustre: 26472:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 789.258810] Lustre: 26472:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 789.265142] Lustre: 26472:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 789.273971] Lustre: 26472:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 789.282214] Lustre: 26472:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 789.290234] Lustre: 26472:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 841.199138] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 841.210256] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 841.231747] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 845.574738] Lustre: server umount lustre-OST07c6 complete [ 855.369689] Lustre: server umount lustre-MDT0000 complete [ 876.559938] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 880.535935] Key type lgssc unregistered [ 880.822941] LNet: 28743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.834629] LNetError: 28743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.853297] LNet: Removed LNI 192.168.204.136@tcp [ 881.992684] Key type .llcrypt unregistered [ 882.006897] Key type ._llcrypt unregistered [ 902.148810] Key type ._llcrypt registered [ 902.154095] Key type .llcrypt registered [ 902.318609] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 903.961033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 903.991327] alg: No test for adler32 (adler32-zlib) [ 905.145157] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 905.388216] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 907.152170] Key type lgssc registered [ 908.226455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 916.738563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 916.752675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 918.191388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 922.694844] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 927.715720] LustreError: 30016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 927.731949] LustreError: 30016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 931.380301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 931.765595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 936.641232] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 944.643037] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 950.927857] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 958.418170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 958.905958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 963.022843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 967.652294] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 977.406965] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 993.761749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 993.773562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 993.812327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 994.327344] Lustre: server umount lustre-OST0000 complete [ 998.369885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 998.382529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 998.396801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1001.962117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1001.962698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1004.397335] Lustre: server umount lustre-MDT0000 complete [ 1007.077237] LustreError: 30016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1007.095703] LustreError: 30016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1007.843874] LustreError: 32999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793948 with bad export cookie 10719800712694338576 [ 1007.852143] LustreError: 32999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1007.854662] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1017.524245] Lustre: server umount lustre-MDT0001 complete [ 1028.357053] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 1031.208497] Key type lgssc unregistered [ 1031.548631] LNet: 33756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1031.561844] LNetError: 33756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1031.579352] LNet: Removed LNI 192.168.204.136@tcp [ 1032.671361] Key type .llcrypt unregistered [ 1032.675732] Key type ._llcrypt unregistered [ 1052.217325] Key type ._llcrypt registered [ 1052.220043] Key type .llcrypt registered [ 1052.322945] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1053.746417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1053.836963] alg: No test for adler32 (adler32-zlib) [ 1054.894732] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 1055.227967] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 1056.968356] Key type lgssc registered [ 1058.734758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1067.837837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1067.867196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.503313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.124370] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1079.781083] LustreError: 35028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.811914] LustreError: 35028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1084.824732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1084.897296] LustreError: 35029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1085.198778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1090.888433] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1098.629516] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1104.907128] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1112.672497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1112.952517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1115.010717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1119.949743] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1127.871254] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1139.683772] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1139.689211] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1139.696433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1139.700573] Lustre: Skipped 1 previous similar message [ 1139.712597] Lustre: Skipped 1 previous similar message [ 1144.370841] Lustre: server umount lustre-OST0000 complete [ 1149.927395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1149.932245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1149.939691] Lustre: Skipped 1 previous similar message [ 1149.954356] Lustre: Skipped 1 previous similar message [ 1155.043808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1155.052255] Lustre: Skipped 1 previous similar message [ 1155.646313] Lustre: server umount lustre-MDT0000 complete [ 1160.028751] LustreError: 35008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794100 with bad export cookie 1254410704884984800 [ 1160.035576] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1160.039222] LustreError: 35008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1160.496691] Lustre: server umount lustre-MDT0001 complete [ 1174.879792] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 1178.420725] Key type lgssc unregistered [ 1178.739920] LNet: 38566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1178.756625] LNetError: 38566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1178.772750] LNet: Removed LNI 192.168.204.136@tcp [ 1179.916742] Key type .llcrypt unregistered [ 1179.920581] Key type ._llcrypt unregistered [ 1204.525780] Key type ._llcrypt registered [ 1204.529810] Key type .llcrypt registered [ 1204.646718] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1205.930157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1206.031318] alg: No test for adler32 (adler32-zlib) [ 1207.179485] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 1207.536123] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 1209.184194] Key type lgssc registered [ 1210.664274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1219.698457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1219.723615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1221.293224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1226.025722] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1231.864972] LustreError: 39987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1231.890568] LustreError: 39987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1236.990745] LustreError: 39988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1237.109571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1237.777224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1243.254946] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1252.174918] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1258.938544] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1265.983712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1266.291367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1267.323155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1272.526718] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1280.340761] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1293.404251] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.36@tcp (stopping) [ 1294.315049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1294.315968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1294.322808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1294.334421] Lustre: Skipped 2 previous similar messages [ 1294.350660] Lustre: Skipped 2 previous similar messages [ 1296.519259] Lustre: server umount lustre-MDT0000 complete [ 1298.482689] LustreError: 41826:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1300.010715] LustreError: 39969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794240 with bad export cookie 8759823941730001160 [ 1300.025105] LustreError: 39969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1300.026191] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1303.602923] LustreError: 39984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1303.618266] LustreError: 39984:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1303.656492] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.36@tcp (stopping) [ 1304.546649] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1306.450455] Lustre: server umount lustre-MDT0001 complete [ 1327.556071] Lustre: server umount lustre-OST0000 complete [ 1342.668742] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 1345.054522] Key type lgssc unregistered [ 1345.353814] LNet: 43857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1345.373136] LNetError: 43857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1345.394272] LNet: Removed LNI 192.168.204.136@tcp [ 1346.255959] Key type .llcrypt unregistered [ 1346.260061] Key type ._llcrypt unregistered [ 1368.133604] Key type ._llcrypt registered [ 1368.137137] Key type .llcrypt registered [ 1368.235995] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1369.021256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1369.044161] alg: No test for adler32 (adler32-zlib) [ 1370.103402] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 1370.299464] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 1371.952232] Key type lgssc registered [ 1373.096213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1381.435738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1381.449392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1382.994644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1387.700331] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1393.124048] LustreError: 45275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1393.152921] LustreError: 45275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1396.650504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1396.995220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1401.966153] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1410.416626] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1418.304414] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1426.677099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1427.140260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1428.232347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1434.291768] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1442.914966] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1451.109257] Lustre: lustre-OST0000: Not available for connect from 192.168.204.36@tcp (stopping) [ 1452.010045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1452.012921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1452.025379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1452.057066] Lustre: Skipped 1 previous similar message [ 1456.188652] Lustre: lustre-OST0000: Not available for connect from 192.168.204.36@tcp (stopping) [ 1456.195281] Lustre: Skipped 1 previous similar message [ 1457.017940] Lustre: server umount lustre-OST0000 complete [ 1476.581482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1476.594331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1476.614059] Lustre: Skipped 1 previous similar message [ 1476.634272] Lustre: Skipped 1 previous similar message [ 1481.698345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1481.713750] Lustre: Skipped 1 previous similar message [ 1482.559914] Lustre: server umount lustre-MDT0000 complete [ 1486.624670] LustreError: 46012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794427 with bad export cookie 5644107308967057029 [ 1486.625183] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1486.637788] LustreError: 46012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1493.127535] Lustre: server umount lustre-MDT0001 complete [ 1505.169318] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 1507.936929] Key type lgssc unregistered [ 1508.198688] LNet: 48987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1508.210365] LNetError: 48987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1508.233333] LNet: Removed LNI 192.168.204.136@tcp [ 1508.947887] Key type .llcrypt unregistered [ 1508.952501] Key type ._llcrypt unregistered [ 1534.711725] Key type ._llcrypt registered [ 1534.714319] Key type .llcrypt registered [ 1534.839505] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1535.887425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1535.956184] alg: No test for adler32 (adler32-zlib) [ 1537.045130] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 1537.257988] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 1539.024177] Key type lgssc registered [ 1540.596767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1549.362867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1549.396267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1551.081194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1557.078699] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1560.547649] LustreError: 50404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1560.562036] LustreError: 50404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1565.684838] LustreError: 50403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1567.673791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1568.002518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1572.994916] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1581.475542] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1588.540221] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1595.531868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1595.889280] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1596.045850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1601.520374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1602.907182] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1612.521955] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1620.151245] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1624.309909] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1636.709516] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2621h36m14s [ 1639.054950] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1643.495710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1643.499203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1643.508627] Lustre: Skipped 1 previous similar message [ 1643.524694] Lustre: Skipped 2 previous similar messages [ 1644.581181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1644.587918] Lustre: Skipped 1 previous similar message [ 1647.216537] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.36@tcp (stopping) [ 1647.224354] Lustre: Skipped 1 previous similar message [ 1647.875221] Lustre: server umount lustre-MDT0000 complete [ 1650.659191] LustreError: 50398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1650.684695] LustreError: 50398:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1651.924841] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1653.304246] LustreError: 50399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1659.282418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1659.420982] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1659.457898] LustreError: 50398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1659.488114] LustreError: 50398:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1659.717832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1659.795606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1660.729732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1665.598534] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1669.620232] Lustre: DEBUG MARKER: Wait for df (36733) ... [ 1671.641913] Lustre: DEBUG MARKER: done [ 1679.329196] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1679.340735] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1679.358965] Lustre: Skipped 1 previous similar message [ 1679.376840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1680.356050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1680.370355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1682.594555] Lustre: server umount lustre-OST0000 complete [ 1691.632218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1691.632635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1691.648683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1691.666383] Lustre: Skipped 1 previous similar message [ 1691.676977] Lustre: Skipped 3 previous similar messages [ 1692.982903] Lustre: server umount lustre-MDT0000 complete [ 1697.233031] LustreError: 53031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794637 with bad export cookie 17957667704427252888 [ 1697.237077] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1697.252476] LustreError: 53031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1697.743954] Lustre: server umount lustre-MDT0001 complete [ 1712.030609] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 1715.495650] Key type lgssc unregistered [ 1715.881812] LNet: 55332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1715.891477] LNetError: 55332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1715.920906] LNet: Removed LNI 192.168.204.136@tcp [ 1717.210500] Key type .llcrypt unregistered [ 1717.213894] Key type ._llcrypt unregistered [ 1763.308276] Key type ._llcrypt registered [ 1763.311586] Key type .llcrypt registered [ 1763.471900] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1765.137934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1765.163058] alg: No test for adler32 (adler32-zlib) [ 1766.208701] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 1766.504745] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 1768.264329] Key type lgssc registered [ 1769.673575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1778.900746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1778.927963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1780.247419] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1780.288369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1780.519753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1786.091249] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1795.246787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1795.351644] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1795.380447] Lustre: 57542:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1795.635404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1800.333950] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1808.118350] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1813.889175] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1821.630298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1821.962404] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1822.175538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1829.461765] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1830.635076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1838.106494] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1844.916785] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2621h39m42s [ 1846.508500] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1852.814149] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1855.586612] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1855.792723] LustreError: 57549:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1876.098049] Lustre: lustre-MDT0000: Client aca67b64-b1f4-4c94-aedd-78ef74fd67b8 (at 192.168.204.36@tcp) reconnecting [ 1895.832159] LustreError: 57549:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1898.654383] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1901.917073] Lustre: DEBUG MARKER: done [ 1910.757641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1910.763523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1910.780899] Lustre: Skipped 1 previous similar message [ 1910.794433] Lustre: Skipped 1 previous similar message [ 1914.804554] Lustre: server umount lustre-OST0000 complete [ 1920.994536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1921.000125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1921.025620] Lustre: Skipped 1 previous similar message [ 1921.045110] Lustre: Skipped 1 previous similar message [ 1922.084980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1922.096820] Lustre: Skipped 1 previous similar message [ 1924.986422] Lustre: server umount lustre-MDT0000 complete [ 1929.183290] LustreError: 57535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794869 with bad export cookie 7896385880581556781 [ 1929.184863] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1929.210033] LustreError: 57535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1929.868888] Lustre: server umount lustre-MDT0001 complete [ 1940.721317] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 1943.014470] Key type lgssc unregistered [ 1943.234483] LNet: 61877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1943.240434] LNetError: 61877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1943.251592] LNet: Removed LNI 192.168.204.136@tcp [ 1943.927770] Key type .llcrypt unregistered [ 1943.929576] Key type ._llcrypt unregistered [ 1984.140622] Key type ._llcrypt registered [ 1984.144725] Key type .llcrypt registered [ 1984.255874] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1985.274503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1985.331369] alg: No test for adler32 (adler32-zlib) [ 1986.376853] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 1986.572933] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 1988.256175] Key type lgssc registered [ 1989.442046] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1992.760583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1992.797230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1994.124195] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1994.150709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1994.325478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1995.279598] Lustre: Failing over lustre-MDT0000 [ 1995.609825] Lustre: server umount lustre-MDT0000 complete [ 2002.050406] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 21:42:20 (1787794940) [ 2008.920681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2009.489391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2014.405908] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2023.323630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2023.545085] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2023.572670] Lustre: 64674:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2023.843187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2029.383506] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2037.915154] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2044.608690] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2052.128337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2052.326369] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2052.520846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2058.398968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2060.265199] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2067.765766] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2074.077377] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2084.322773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2084.330435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2084.339740] Lustre: Skipped 2 previous similar messages [ 2084.354016] Lustre: Skipped 2 previous similar messages [ 2087.492820] Lustre: server umount lustre-MDT0000 complete [ 2091.486598] LustreError: 65420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795032 with bad export cookie 15128709866084273115 [ 2091.491465] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2091.501375] LustreError: 65420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2092.090658] Lustre: server umount lustre-MDT0001 complete [ 2095.739700] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2100.674357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2109.646293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2110.944237] Lustre: 63598:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787795035/real 1787795035] req@00000000ac0af192 x1874638853416960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787795051 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2110.972954] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2112.800130] Lustre: 63597:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787795037/real 1787795037] req@00000000d7371927 x1874638853417088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787795053 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2116.064597] Lustre: 63600:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787795040/real 1787795040] req@00000000f2cabc2e x1874638853417216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787795056 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2117.572565] LustreError: 66526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2117.603969] LustreError: 66526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2117.693380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2117.813210] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2118.688215] Lustre: 63599:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787795042/real 1787795042] req@0000000058057446 x1874638853417472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787795058 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2120.741386] LustreError: 66526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2121.763489] LustreError: 68446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2121.785879] LustreError: 68446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2122.784092] Lustre: 63598:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787795047/real 1787795047] req@00000000139c47a7 x1874638853417600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787795063 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2122.843076] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2126.829485] LustreError: 68445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2126.852791] LustreError: 68445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2131.792274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2131.946581] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2131.960306] Lustre: Skipped 1 previous similar message [ 2132.165713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2136.461557] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2137.577346] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2139.619209] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2144.742048] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2152.116711] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2160.101613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2160.109471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2160.126166] Lustre: Skipped 2 previous similar messages [ 2160.143117] Lustre: Skipped 2 previous similar messages [ 2165.220607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2165.238032] Lustre: Skipped 2 previous similar messages [ 2165.530692] Lustre: server umount lustre-MDT0000 complete [ 2169.323198] LustreError: 64667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795110 with bad export cookie 15128709866084275390 [ 2169.327821] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2169.343024] LustreError: 64667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2169.687395] Lustre: server umount lustre-MDT0001 complete [ 2176.490220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2186.409345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2186.720949] Lustre: 63597:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787795111/real 1787795111] req@0000000057d52f55 x1874638853446144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787795127 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2186.762366] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2194.976951] LustreError: 63596:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f63aad7a x1874638853447168/t0(0) o250->MGC192.168.204.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2195.417125] LustreError: 66526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2195.526907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2195.608371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2199.661886] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2206.695210] LustreError: 69501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2206.719385] LustreError: 69501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2208.126499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2213.262450] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2213.875514] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2215.985535] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2220.856332] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2227.164287] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2236.387025] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2236.388354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2236.414383] Lustre: Skipped 2 previous similar messages [ 2236.430813] Lustre: Skipped 1 previous similar message [ 2242.745529] Lustre: server umount lustre-MDT0000 complete [ 2246.627875] LustreError: 71132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2246.638251] LustreError: 71132:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2246.702548] LustreError: 65420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795187 with bad export cookie 15128709866084276769 [ 2246.713375] LustreError: 65420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2246.717938] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2246.990148] Lustre: server umount lustre-MDT0001 complete [ 2253.455630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2256.840703] Lustre: DEBUG MARKER: files compared the same [ 2269.298249] Lustre: server umount lustre-OST0000 complete [ 2285.799482] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 2288.672563] Key type lgssc unregistered [ 2288.938852] LNet: 74409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2288.943123] LNetError: 74409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2288.958621] LNet: Removed LNI 192.168.204.136@tcp [ 2289.750286] Key type .llcrypt unregistered [ 2289.754680] Key type ._llcrypt unregistered [ 2314.038654] Key type ._llcrypt registered [ 2314.046693] Key type .llcrypt registered [ 2314.160978] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2315.298636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2315.436378] alg: No test for adler32 (adler32-zlib) [ 2316.517811] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 2316.782984] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 2318.480168] Key type lgssc registered [ 2319.632225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2327.068369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2327.098680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2328.549615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2332.708467] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug +malloc all [ 2338.794496] LustreError: 75827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2338.811146] LustreError: 75827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2341.214110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2341.513212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2346.114149] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug +malloc all [ 2353.540763] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2359.142764] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2367.196404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2367.488355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2373.622057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2375.093648] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug +malloc all [ 2384.278951] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2397.159611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2397.167439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2397.172745] Lustre: Skipped 1 previous similar message [ 2397.186717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2401.659190] Lustre: server umount lustre-OST0000 complete [ 2407.392992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2407.397110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2407.407933] Lustre: Skipped 1 previous similar message [ 2407.422364] Lustre: Skipped 2 previous similar messages [ 2412.515103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2412.523522] Lustre: Skipped 1 previous similar message [ 2417.632879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2417.643945] Lustre: Skipped 1 previous similar message [ 2420.192872] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2420.489930] Lustre: server umount lustre-MDT0000 complete [ 2422.755287] LustreError: 75823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2422.774744] LustreError: 75823:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2424.389653] LustreError: 75809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795365 with bad export cookie 7874212826997259294 [ 2424.404256] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2424.725759] Lustre: server umount lustre-MDT0001 complete [ 2438.621338] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 2442.026452] Key type lgssc unregistered [ 2442.374883] LNet: 79364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2442.384358] LNetError: 79364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2442.406686] LNet: Removed LNI 192.168.204.136@tcp [ 2443.424859] Key type .llcrypt unregistered [ 2443.428305] Key type ._llcrypt unregistered [ 2468.304382] Key type ._llcrypt registered [ 2468.308610] Key type .llcrypt registered [ 2468.503508] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2469.839734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2469.875721] alg: No test for adler32 (adler32-zlib) [ 2470.981547] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 2471.226222] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 2472.953037] Key type lgssc registered [ 2474.210698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2482.618547] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2482.634829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2498.146552] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2503.267107] LustreError: 80759:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.136@tcp: failed processing log, type 4: rc = -110 [ 2529.952279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2536.397515] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2543.756102] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2551.760036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2551.903489] Lustre: *** cfs_fail_loc=706, val=0*** [ 2552.302483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2552.375260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2557.206513] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2557.423725] LustreError: 80784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2557.445234] LustreError: 80784:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2562.533733] LustreError: 80785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.612040] LustreError: 80784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.629068] LustreError: 80784:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2566.504528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2566.990931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2571.519476] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2578.598329] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2584.622921] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2587.627047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2587.629429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2587.658122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2592.740631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2592.751121] Lustre: Skipped 1 previous similar message [ 2593.316622] Lustre: server umount lustre-OST0000 complete [ 2597.857803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2597.876281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2597.879303] Lustre: Skipped 2 previous similar messages [ 2597.911114] Lustre: Skipped 1 previous similar message [ 2602.984356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2603.000920] Lustre: Skipped 1 previous similar message [ 2604.185851] Lustre: server umount lustre-MDT0000 complete [ 2608.099755] LustreError: 81845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2608.121395] LustreError: 81845:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2608.422058] LustreError: 83623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795549 with bad export cookie 16218363042366620768 [ 2608.429819] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2608.443048] LustreError: 83623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2608.896265] Lustre: server umount lustre-MDT0001 complete [ 2620.581743] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 2623.059256] Key type lgssc unregistered [ 2623.376676] LNet: 84379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2623.384788] LNetError: 84379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2623.401948] LNet: Removed LNI 192.168.204.136@tcp [ 2624.171435] Key type .llcrypt unregistered [ 2624.178896] Key type ._llcrypt unregistered [ 2652.321405] Key type ._llcrypt registered [ 2652.323096] Key type .llcrypt registered [ 2652.480701] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2653.818101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2653.856962] alg: No test for adler32 (adler32-zlib) [ 2655.468409] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 2655.754253] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 2657.464168] Key type lgssc registered [ 2658.352439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2668.090375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2668.126251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2673.296848] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2681.913784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2682.435302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2686.553846] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2687.468547] LustreError: 86481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2687.491961] LustreError: 86481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2692.577375] LustreError: 86485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2692.601647] LustreError: 86485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2695.967684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2696.280349] LustreError: 86481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2696.309437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2696.316626] LustreError: 86481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2701.798217] LustreError: 86484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2701.816589] LustreError: 86484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2702.682068] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2706.919957] LustreError: 86485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2706.936100] LustreError: 86485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2711.838804] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2711.864963] Lustre: MGS: Client a60524b9-a4da-4c90-913f-0635ca29ed00 (at 0@lo) reconnecting [ 2711.891756] LustreError: 86495:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2712.116653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2712.186298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2719.550516] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2737.636706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2737.645456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2737.657582] Lustre: Skipped 1 previous similar message [ 2737.657882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2737.693183] Lustre: Skipped 1 previous similar message [ 2742.759188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2742.770921] Lustre: Skipped 1 previous similar message [ 2743.001526] Lustre: server umount lustre-OST0000 complete [ 2747.875524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2747.877653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2747.898598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2752.572681] Lustre: server umount lustre-MDT0000 complete [ 2752.994483] LustreError: 86485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2753.018369] LustreError: 86485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2757.086093] Lustre: server umount lustre-MDT0001 complete [ 2761.109801] Lustre: server umount MGS complete [ 2773.354157] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 21:55:11 (1787795711) [ 2806.419466] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 2816.179673] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2857.369116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2862.300532] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2873.662880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2873.997953] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2874.069001] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2874.207958] Lustre: lustre-MDT0001: new disk, initializing [ 2874.330498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2874.392734] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2874.410849] Lustre: 93676:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2874.432971] LustreError: 93676:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2880.008832] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2892.152833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2892.425715] Lustre: lustre-OST0000: new disk, initializing [ 2892.438378] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2892.457391] Lustre: 94441:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2892.570773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2899.822347] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2902.562095] Lustre: 94887:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000026491e65 x1874639554913664/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787795859 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2912.339539] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2922.146789] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2922.163694] Lustre: MGS: Client 310434f8-ab4d-4f3e-86f1-b2717120a7a7 (at 0@lo) reconnecting [ 2922.222596] Lustre: 95488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2922.237383] LustreError: 93662:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2922.254244] LustreError: 93662:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2922.272775] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2922.408577] Lustre: lustre-MDT0000: new disk, initializing [ 2922.605342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2922.642272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2922.684440] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2923.748988] Lustre: 95513:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000e2196678 x1874639554920960/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787795880 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2928.933750] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2930.515130] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2932.841946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2932.846515] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2933.010275] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2957.283663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2957.300393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2957.302792] Lustre: Skipped 2 previous similar messages [ 2957.337784] Lustre: Skipped 2 previous similar messages [ 2962.419504] Lustre: server umount lustre-OST0000 complete [ 2967.525383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2967.542529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2967.549065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2967.566287] Lustre: Skipped 1 previous similar message [ 2967.580426] Lustre: Skipped 1 previous similar message [ 2977.771967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2977.796674] Lustre: Skipped 3 previous similar messages [ 2981.345354] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2981.507851] Lustre: server umount lustre-MDT0000 complete [ 2982.884714] LustreError: 95946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2982.909616] LustreError: 95946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2992.794698] Lustre: server umount lustre-MDT0001 complete [ 2996.932888] Lustre: server umount MGS complete [ 3010.519403] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 21:59:08 (1787795948) [ 3031.827505] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 3035.041773] Key type lgssc unregistered [ 3035.461578] LNet: 98014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3035.478638] LNetError: 98014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3035.506442] LNet: Removed LNI 192.168.204.136@tcp [ 3036.663351] Key type .llcrypt unregistered [ 3036.666914] Key type ._llcrypt unregistered [ 3057.248747] Key type ._llcrypt registered [ 3057.251441] Key type .llcrypt registered [ 3057.415036] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3058.605171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3058.653968] alg: No test for adler32 (adler32-zlib) [ 3059.635380] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 3059.838442] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 3061.504267] Key type lgssc registered [ 3062.759031] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3072.091675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3072.152056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3073.749255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3079.248180] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3084.265957] LustreError: 99236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3084.288335] LustreError: 99236:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3089.251659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3089.377528] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3089.551116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3094.380307] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3102.733066] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3108.798855] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3117.365847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3117.807365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3119.856115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3125.265849] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3134.382927] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3145.698660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3145.701321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3145.710329] LustreError: Skipped 1 previous similar message [ 3145.735755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3148.897771] Lustre: server umount lustre-OST0000 complete [ 3155.953113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3155.956201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3155.982914] Lustre: Skipped 2 previous similar messages [ 3156.007992] Lustre: Skipped 2 previous similar messages [ 3159.632294] Lustre: server umount lustre-MDT0000 complete [ 3161.063145] LustreError: 99231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3161.085890] LustreError: 99231:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3164.025575] LustreError: 99218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796104 with bad export cookie 7035171489978261187 [ 3164.029822] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3164.041080] LustreError: 99218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3164.573101] Lustre: server umount lustre-MDT0001 complete [ 3178.287226] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 3181.444836] Key type lgssc unregistered [ 3181.835837] LNet: 102787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3181.845870] LNetError: 102787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3181.879553] LNet: Removed LNI 192.168.204.136@tcp [ 3182.804326] Key type .llcrypt unregistered [ 3182.807879] Key type ._llcrypt unregistered [ 3200.644213] Key type ._llcrypt registered [ 3200.646131] Key type .llcrypt registered [ 3200.734189] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3201.979502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3202.038276] alg: No test for adler32 (adler32-zlib) [ 3203.182722] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 3203.443036] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 3205.128191] Key type lgssc registered [ 3206.336471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3213.200758] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3213.209800] LustreError: 103872:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3213.214599] Lustre: Skipped 1 previous similar message [ 3213.220046] LustreError: 103871:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3213.235281] LustreError: 103872:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4974 [ 3213.241987] LustreError: 103872:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3213.248420] LustreError: 103872:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3213.263178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3214.677732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3222.823119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3223.243222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3228.319661] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3234.499989] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing lsmod [ 3237.718451] LustreError: 105071:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3237.764804] LustreError: 105078:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3237.770153] LustreError: 105071:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4953 [ 3237.777300] LustreError: 105071:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3237.783537] LustreError: 105071:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3237.787230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3238.198720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3240.252816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3244.004273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3244.016050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3244.030677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3245.539863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3245.552744] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3245.568977] Lustre: Skipped 1 previous similar message [ 3247.656384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3247.668767] Lustre: Skipped 4 previous similar messages [ 3247.873799] Lustre: server umount lustre-MDT0000 complete [ 3252.386416] LustreError: 105264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796193 with bad export cookie 1113633138985502870 [ 3252.393350] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3252.401124] LustreError: 105264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3252.705124] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3252.705792] LustreError: 104403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3252.729514] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3252.761307] LustreError: 104403:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3253.856719] LustreError: 104340:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b8de2c47 x1874640129513984/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3254.036720] Lustre: server umount lustre-MDT0001 complete [ 3258.012914] Lustre: server umount lustre-OST0000 complete [ 3267.399841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3268.005307] LustreError: 106141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3268.108650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3272.567869] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3273.186703] LustreError: 106142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3277.287624] LustreError: 106141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3281.755601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3282.299073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3286.728650] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3295.849857] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3303.182768] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3310.278656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3310.816432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3312.906662] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3318.293475] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3327.714562] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3336.681946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3336.689125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3336.700813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3336.702341] Lustre: Skipped 1 previous similar message [ 3336.722979] Lustre: Skipped 1 previous similar message [ 3340.559508] Lustre: server umount lustre-OST0000 complete [ 3346.916769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3346.928188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3346.937889] Lustre: Skipped 1 previous similar message [ 3350.753858] Lustre: server umount lustre-MDT0000 complete [ 3352.036519] LustreError: 106136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3352.060159] LustreError: 106136:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3354.632588] LustreError: 106123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796295 with bad export cookie 1113633138985503794 [ 3354.646088] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3354.649129] LustreError: 106123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3355.078907] Lustre: server umount lustre-MDT0001 complete [ 3368.751754] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 3372.243119] Key type lgssc unregistered [ 3372.706885] LNet: 109677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3372.717147] LNetError: 109677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3372.756204] LNet: Removed LNI 192.168.204.136@tcp [ 3373.766278] Key type .llcrypt unregistered [ 3373.771055] Key type ._llcrypt unregistered [ 3406.761490] Key type ._llcrypt registered [ 3406.763871] Key type .llcrypt registered [ 3407.031414] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3408.407175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3408.435626] alg: No test for adler32 (adler32-zlib) [ 3409.748779] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 3410.256218] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 3412.072182] Key type lgssc registered [ 3414.201534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3423.796557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3423.834050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3425.376217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3430.196332] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3435.495382] LustreError: 111100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3435.516641] LustreError: 111100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3440.611511] LustreError: 111101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3440.826368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3441.360816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3446.545723] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3455.150633] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3462.166521] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3470.011519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3470.322822] LustreError: 112921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3470.524756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3475.969589] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3477.610830] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3486.915716] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3492.458435] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3496.476862] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3501.537079] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3501.543891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3501.550371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3501.565996] Lustre: Skipped 1 previous similar message [ 3506.672453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3506.686227] Lustre: Skipped 2 previous similar messages [ 3507.006284] Lustre: server umount lustre-OST0000 complete [ 3513.314759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3513.329266] LustreError: Skipped 1 previous similar message [ 3513.343387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3513.374777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3516.897365] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3516.897707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3518.730828] Lustre: server umount lustre-MDT0000 complete [ 3522.032099] LustreError: 111100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3522.055507] LustreError: 111100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3523.261539] LustreError: 111080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796464 with bad export cookie 3757354402571027260 [ 3523.266324] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3523.276926] LustreError: 111080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3523.690232] Lustre: server umount lustre-MDT0001 complete [ 3538.840375] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 3542.535146] Key type lgssc unregistered [ 3542.849845] LNet: 114740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3542.855789] LNetError: 114740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3542.874828] LNet: Removed LNI 192.168.204.136@tcp [ 3543.913739] Key type .llcrypt unregistered [ 3543.916375] Key type ._llcrypt unregistered [ 3561.373091] Key type ._llcrypt registered [ 3561.379082] Key type .llcrypt registered [ 3561.486117] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3562.627264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3562.638063] alg: No test for adler32 (adler32-zlib) [ 3563.804588] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 3564.098120] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 3565.792182] Key type lgssc registered [ 3566.879247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3575.177827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3575.205821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3576.769657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3581.875939] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3587.046280] LustreError: 115963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3587.064143] LustreError: 115963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3591.430947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3592.014788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3598.001742] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3606.494819] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3613.218615] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3620.661181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3621.123893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3622.131918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3628.309838] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3636.883585] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3646.960729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3646.969560] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3646.990118] Lustre: Skipped 1 previous similar message [ 3646.990910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3651.951334] Lustre: server umount lustre-OST0000 complete [ 3657.192742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3657.208814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3657.215599] Lustre: Skipped 1 previous similar message [ 3657.228179] Lustre: Skipped 2 previous similar messages [ 3662.308978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3662.323923] Lustre: Skipped 1 previous similar message [ 3662.834589] Lustre: server umount lustre-MDT0000 complete [ 3666.405072] LustreError: 115958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3666.425655] LustreError: 115958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3666.542251] LustreError: 115944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796607 with bad export cookie 1643304474099529317 [ 3666.564561] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3666.569782] LustreError: 115944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3666.862972] Lustre: server umount lustre-MDT0001 complete [ 3678.931458] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 3681.875291] Key type lgssc unregistered [ 3682.226272] LNet: 119500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3682.241294] LNetError: 119500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3682.260864] LNet: Removed LNI 192.168.204.136@tcp [ 3683.138894] Key type .llcrypt unregistered [ 3683.155939] Key type ._llcrypt unregistered [ 3709.275624] Key type ._llcrypt registered [ 3709.278239] Key type .llcrypt registered [ 3709.412147] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3710.782921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3710.801070] alg: No test for adler32 (adler32-zlib) [ 3711.998927] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 3712.271640] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 3714.006439] Key type lgssc registered [ 3715.167487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3724.361230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3724.381884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3725.869650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3730.267077] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3736.038973] LustreError: 120919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3736.060022] LustreError: 120919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3738.590284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3739.150905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3744.578813] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3754.296316] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3760.452859] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3768.497405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3769.010917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3771.067377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3776.800519] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3786.579814] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3793.519891] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3797.563617] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3797.575949] Lustre: Skipped 2 previous similar messages [ 3802.468351] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3802.471848] Lustre: Skipped 2 previous similar messages [ 3803.248249] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3803.294233] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3812.556785] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.36@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3812.571736] Lustre: Skipped 2 previous similar messages [ 3822.680877] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.36@tcp 192.168.0.[2,10]@tcp [ 3828.358931] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.36/24@tcp in log lustre-MDT0000 [ 3828.373350] Lustre: Skipped 2 previous similar messages [ 3830.520416] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.36/24@tcp [ 3830.528858] Lustre: Skipped 1 previous similar message [ 3843.045388] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3843.054846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3843.078189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3845.106438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3845.139423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3845.149627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3848.549894] Lustre: server umount lustre-OST0000 complete [ 3855.334854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3855.350601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3855.360978] Lustre: Skipped 1 previous similar message [ 3855.371231] Lustre: Skipped 2 previous similar messages [ 3859.414364] Lustre: server umount lustre-MDT0000 complete [ 3860.464801] LustreError: 122120:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3860.486440] LustreError: 122120:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3863.329322] LustreError: 120900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796804 with bad export cookie 2346595714094336899 [ 3863.342440] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3863.348448] LustreError: 120900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3863.677864] Lustre: server umount lustre-MDT0001 complete [ 3876.389817] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 3879.483133] Key type lgssc unregistered [ 3879.779032] LNet: 125760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3879.791673] LNetError: 125760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3879.840859] LNet: Removed LNI 192.168.204.136@tcp [ 3880.771471] Key type .llcrypt unregistered [ 3880.779149] Key type ._llcrypt unregistered [ 3913.930509] Key type ._llcrypt registered [ 3913.934144] Key type .llcrypt registered [ 3914.086300] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3915.213946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3915.320147] alg: No test for adler32 (adler32-zlib) [ 3916.391120] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 3916.680233] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 3918.417819] Key type lgssc registered [ 3919.815448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3928.720517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3928.745508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3930.256344] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3930.314414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3930.319934] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.36/24@tcp [ 3935.572060] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3939.811929] LustreError: 127324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3939.832657] LustreError: 127324:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3944.931130] LustreError: 127325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3945.426669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3945.743688] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3945.770781] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.36/24@tcp [ 3945.879470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3951.698506] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3961.970272] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3968.825087] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3976.961764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3977.510048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3985.220552] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3995.269784] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4000.051889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4010.466343] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4010.468759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4010.484082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4010.501751] Lustre: Skipped 1 previous similar message [ 4010.515081] Lustre: Skipped 1 previous similar message [ 4012.197051] Lustre: server umount lustre-OST0000 complete [ 4017.634130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4017.645809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4017.663818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4020.712769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4020.721092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4020.740903] Lustre: Skipped 1 previous similar message [ 4022.763805] Lustre: server umount lustre-MDT0000 complete [ 4025.825222] LustreError: 129822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4025.848113] LustreError: 129822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4027.031329] LustreError: 127304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796967 with bad export cookie 12643967771870835383 [ 4027.041882] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4027.046803] LustreError: 127304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4027.755797] Lustre: server umount lustre-MDT0001 complete [ 4041.713978] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 4045.130927] Key type lgssc unregistered [ 4045.469173] LNet: 130918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4045.480776] LNetError: 130918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4045.504298] LNet: Removed LNI 192.168.204.136@tcp [ 4046.576150] Key type .llcrypt unregistered [ 4046.578667] Key type ._llcrypt unregistered [ 4079.329384] Key type ._llcrypt registered [ 4079.336908] Key type .llcrypt registered [ 4079.508954] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 4097.650673] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4098.902715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4099.290210] alg: No test for adler32 (adler32-zlib) [ 4100.372427] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 4100.644266] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 4102.352220] Key type lgssc registered [ 4103.669592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4144.536316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4144.615541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4146.070324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4146.100505] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4146.202474] Lustre: lustre-MDT0000: new disk, initializing [ 4146.288879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4146.308911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4151.107509] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4163.049397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4163.288865] Lustre: 134319:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4163.396839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4163.404162] Lustre: Skipped 1 previous similar message [ 4163.544488] Lustre: lustre-MDT0001: new disk, initializing [ 4163.769614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4163.832991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4163.855409] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4169.023639] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4179.795232] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4185.318141] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4193.105915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4193.386364] Lustre: lustre-OST0000: new disk, initializing [ 4193.393946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4193.405463] Lustre: 136289:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4193.544283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4199.529341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4199.550698] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4199.757365] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4200.098741] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4211.266292] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4217.317432] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4217.595434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4223.115683] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4223.407874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4230.637757] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4230.640649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4230.659218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4230.670801] LustreError: Skipped 1 previous similar message [ 4230.727195] Lustre: Skipped 1 previous similar message [ 4235.752296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4235.776850] Lustre: Skipped 2 previous similar messages [ 4239.840158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4240.051522] Lustre: server umount lustre-OST0000 complete [ 4245.477102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4245.487554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4245.508452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4251.105552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4251.119785] Lustre: Skipped 2 previous similar messages [ 4251.643769] Lustre: server umount lustre-MDT0000 complete [ 4256.124025] LustreError: 135124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797196 with bad export cookie 6751065106835775254 [ 4256.127247] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4256.142061] LustreError: 135124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4256.652946] Lustre: server umount lustre-MDT0001 complete [ 4269.119615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4269.868982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4275.170578] LustreError: 138510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4275.187199] LustreError: 138510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4275.242796] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4279.270340] LustreError: 138511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4284.393905] LustreError: 138510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4286.221577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4286.735875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4292.334252] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4302.849747] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4309.437774] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4317.775060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4318.192359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4325.570192] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4333.675312] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4339.553536] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4340.875530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4346.541746] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4346.788823] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4357.504218] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4357.796364] Lustre: lustre-OST0001: new disk, initializing [ 4357.802800] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4357.811484] Lustre: 141891:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4357.958652] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4366.472152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4366.487737] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4366.642034] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4367.001601] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4376.193286] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4376.387171] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4380.750477] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4380.954647] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4387.105834] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa0d704e0a800.ost_server_uuid 50 [ 4388.948579] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa0d704e0a800.ost_server_uuid in IDLE state after 0 sec [ 4402.664992] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4402.676335] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4402.687678] Lustre: Skipped 1 previous similar message [ 4402.692843] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4402.716285] Lustre: Skipped 1 previous similar message [ 4405.096821] Lustre: server umount lustre-OST0001 complete [ 4407.788895] LustreError: 140334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.814210] LustreError: 140334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4411.362859] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4411.374650] LustreError: Skipped 1 previous similar message [ 4411.381495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4411.393552] Lustre: Skipped 1 previous similar message [ 4411.402893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4412.903077] LustreError: 140334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4412.930396] LustreError: 140334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4415.496689] Lustre: server umount lustre-OST0000 complete [ 4420.065503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4420.071285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4420.083840] Lustre: Skipped 1 previous similar message [ 4428.260248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4428.272676] Lustre: Skipped 6 previous similar messages [ 4433.889038] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4434.124259] Lustre: server umount lustre-MDT0000 complete [ 4437.473565] LustreError: 138507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4437.500081] LustreError: 138507:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4438.151673] LustreError: 141890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797378 with bad export cookie 6751065106835776136 [ 4438.164818] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4438.175565] LustreError: 141890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4438.603905] Lustre: server umount lustre-MDT0001 complete [ 4460.474649] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 4463.997198] Key type lgssc unregistered [ 4464.410767] LNet: 144901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4464.429237] LNetError: 144901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4464.459900] LNet: Removed LNI 192.168.204.136@tcp [ 4465.615563] Key type .llcrypt unregistered [ 4465.617667] Key type ._llcrypt unregistered [ 4512.401623] Key type ._llcrypt registered [ 4512.414784] Key type .llcrypt registered [ 4512.651231] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 4529.376531] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4530.680712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4530.702396] alg: No test for adler32 (adler32-zlib) [ 4531.794801] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 4532.023995] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 4533.696364] Key type lgssc registered [ 4535.175300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4580.303194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4580.374468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4581.593528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4581.643937] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4581.825410] Lustre: lustre-MDT0000: new disk, initializing [ 4581.923689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4581.963860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4588.872634] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4602.885249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4603.044803] Lustre: 149057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4603.126862] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4603.131849] Lustre: Skipped 1 previous similar message [ 4603.250589] Lustre: lustre-MDT0001: new disk, initializing [ 4603.505762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4603.608891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4603.642442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4609.397565] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4620.891513] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4628.029452] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4636.714353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4637.115674] Lustre: lustre-OST0000: new disk, initializing [ 4637.122279] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4637.134049] Lustre: 151029:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4637.281075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4644.474825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4644.488155] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4644.615325] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4645.453904] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4657.515792] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4663.097591] Lustre: Failing over lustre-OST0000 [ 4663.263145] Lustre: server umount lustre-OST0000 complete [ 4680.673843] Lustre: 147283:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787797605/real 1787797605] req@00000000febec6cc x1874641522035840/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787797621 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4680.708966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4681.696146] Lustre: 147284:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787797606/real 1787797606] req@00000000530a1165 x1874641522036864/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787797622 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4681.726639] Lustre: 147284:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4681.739948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4683.739446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4684.215301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4684.242236] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4686.241108] Lustre: 147284:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787797611/real 1787797611] req@00000000186995d8 x1874641522037888/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787797627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4686.285783] Lustre: 147284:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4690.354443] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4691.363572] Lustre: 147283:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787797616/real 1787797616] req@00000000a40314f7 x1874641522038912/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787797632 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4691.404120] Lustre: 147283:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4691.475553] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4691.475885] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4692.112189] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4696.548347] Lustre: 147282:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787797621/real 1787797621] req@000000000ded28cf x1874641522039680/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787797637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4696.571753] Lustre: 147282:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4697.193041] Lustre: Failing over lustre-MDT0000 [ 4697.522993] Lustre: server umount lustre-MDT0000 complete [ 4700.660221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4700.665155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4700.688991] Lustre: Skipped 2 previous similar messages [ 4705.766267] LustreError: 151330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4705.788621] LustreError: 151330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4706.363060] LustreError: 149065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4710.890339] LustreError: 149067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4710.922588] LustreError: 149067:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4716.003965] LustreError: 149070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4716.026456] LustreError: 149070:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4717.027937] Lustre: 147282:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787797641/real 1787797641] req@00000000860b95e4 x1874641522052736/t0(0) o400->MGC192.168.204.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787797657 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4717.068616] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4718.848893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4722.147614] LustreError: 149070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4722.181577] LustreError: 149070:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4726.473557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4726.535165] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4727.860820] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4731.809370] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4731.908622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4731.914592] Lustre: Skipped 1 previous similar message [ 4731.964430] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4732.048470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4742.119360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4742.123622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4742.145983] Lustre: Skipped 1 previous similar message [ 4742.162443] Lustre: Skipped 1 previous similar message [ 4747.237780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4747.249108] Lustre: Skipped 1 previous similar message [ 4748.145491] Lustre: server umount lustre-OST0000 complete [ 4757.474992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4757.475709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4757.518814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4757.527313] Lustre: Skipped 1 previous similar message [ 4759.085478] Lustre: server umount lustre-MDT0000 complete [ 4762.595430] LustreError: 149067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4762.631410] LustreError: 149067:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4764.027267] LustreError: 151028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797704 with bad export cookie 4436503659978893936 [ 4764.034774] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4764.057821] LustreError: 151028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4764.727209] Lustre: server umount lustre-MDT0001 complete [ 4779.512991] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 4783.478900] Key type lgssc unregistered [ 4783.854899] LNet: 154691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4783.869882] LNetError: 154691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4783.896307] LNet: Removed LNI 192.168.204.136@tcp [ 4784.852823] Key type .llcrypt unregistered [ 4784.855485] Key type ._llcrypt unregistered [ 4821.615728] Key type ._llcrypt registered [ 4821.619582] Key type .llcrypt registered [ 4821.727927] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4822.832227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4822.889264] alg: No test for adler32 (adler32-zlib) [ 4824.109510] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 4824.371354] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 4826.072201] Key type lgssc registered [ 4827.475223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4835.844520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4835.858796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4837.452853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4842.929527] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4847.586141] LustreError: 156211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4847.610164] LustreError: 156211:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4852.707123] LustreError: 156210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4853.701094] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4857.830578] LustreError: 156211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4857.933139] LustreError: 156239:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4857.948430] LustreError: 156239:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4857.955561] LustreError: 156239:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4858.360780] Lustre: server umount lustre-MDT0000 complete [ 4875.271460] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 4885.158518] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4928.357443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4928.712531] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4928.751195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4928.871458] Lustre: lustre-MDT0000: new disk, initializing [ 4928.962880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4928.985393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4933.769577] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4945.419393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4945.611526] Lustre: 159546:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4945.683987] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4945.690425] Lustre: Skipped 1 previous similar message [ 4945.772345] Lustre: lustre-MDT0001: new disk, initializing [ 4945.871483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4945.920138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4945.931686] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4950.874356] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4961.786270] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4968.535885] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4975.928631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4976.192767] Lustre: lustre-OST0000: new disk, initializing [ 4976.198139] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4976.207231] Lustre: 161511:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4976.304853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4981.869836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4981.890789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4982.022121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4983.548466] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4993.888338] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4998.402694] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5000.627947] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 22:32:18 (1787797938) [ 5273.568666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5273.582205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5273.601919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5274.091459] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5274.116431] Lustre: Skipped 1 previous similar message [ 5274.132908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5274.144146] Lustre: Skipped 2 previous similar messages [ 5279.200832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5279.200831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5279.211566] Lustre: Skipped 1 previous similar message [ 5284.321916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5284.326537] Lustre: Skipped 2 previous similar messages [ 5285.344264] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5285.544743] Lustre: server umount lustre-MDT0000 complete [ 5289.441653] LustreError: 161798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5289.473695] LustreError: 161798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5290.049958] LustreError: 159535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798230 with bad export cookie 141312532833232777 [ 5290.058790] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5290.062547] LustreError: 159535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5294.560741] LustreError: 162231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5294.561670] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5294.604876] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5296.369854] Lustre: server umount lustre-MDT0001 complete [ 5307.098074] Lustre: server umount lustre-OST0000 complete [ 5319.212847] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 5332.424531] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 5375.055716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5375.354846] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5375.402089] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5375.500272] Lustre: lustre-MDT0000: new disk, initializing [ 5375.571545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5375.587646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5380.686607] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5391.577857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5391.638042] Lustre: 166190:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5391.689767] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5391.695828] Lustre: Skipped 1 previous similar message [ 5391.697624] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5391.704464] Lustre: Skipped 1 previous similar message [ 5391.740822] Lustre: lustre-MDT0001: new disk, initializing [ 5391.862756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5391.913280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5391.925634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5396.472215] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5407.376345] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5414.847817] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5423.290802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5423.615704] Lustre: lustre-OST0000: new disk, initializing [ 5423.620848] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5423.628074] Lustre: 168163:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5423.733264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5431.115394] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5432.915623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5432.924043] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5433.028937] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5441.317092] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5446.647852] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5446.890713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5451.690339] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5451.994418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5458.915199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5458.916570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5458.918407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5458.937872] Lustre: Skipped 1 previous similar message [ 5460.374665] Lustre: server umount lustre-OST0000 complete [ 5468.643812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5468.649424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5470.741911] Lustre: server umount lustre-MDT0000 complete [ 5474.207683] LustreError: 166183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798414 with bad export cookie 141312532833391600 [ 5474.213556] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5474.220588] LustreError: 166183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5474.516810] Lustre: server umount lustre-MDT0001 complete [ 5483.163222] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:40:21 (1787798421) [ 5500.549906] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 5511.270911] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 5547.658728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5547.999421] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5548.060050] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5548.153547] Lustre: lustre-MDT0000: new disk, initializing [ 5548.264935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5548.278826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5554.371668] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5565.008446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5565.184828] Lustre: 172918:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5565.524578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5569.906618] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5578.459310] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5583.751859] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5590.485375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5590.692690] Lustre: lustre-OST0000: new disk, initializing [ 5590.696120] Lustre: Skipped 1 previous similar message [ 5590.703504] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5590.713017] Lustre: Skipped 2 previous similar messages [ 5590.719189] Lustre: 174894:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5597.275488] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5600.883694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5600.893814] Lustre: Skipped 1 previous similar message [ 5600.908209] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5601.041813] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5607.193136] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5616.610751] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5616.611102] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5616.617308] LustreError: Skipped 1 previous similar message [ 5616.621508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5616.636065] Lustre: Skipped 2 previous similar messages [ 5616.658153] Lustre: Skipped 5 previous similar messages [ 5622.010866] Lustre: server umount lustre-OST0000 complete [ 5626.849660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5632.522203] Lustre: server umount lustre-MDT0000 complete [ 5636.258655] LustreError: 172911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798577 with bad export cookie 141312532833392489 [ 5636.262922] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5636.270757] LustreError: 172911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5636.600996] Lustre: server umount lustre-MDT0001 complete [ 5645.417708] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:43:04 (1787798584) [ 5660.983260] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 5670.643349] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 5708.336198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5708.511562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5708.531189] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5708.588988] Lustre: lustre-MDT0000: new disk, initializing [ 5708.661018] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5708.667614] Lustre: Skipped 2 previous similar messages [ 5708.684556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5712.568981] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5721.817291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5721.977753] Lustre: 179229:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5722.234334] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5726.581406] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5735.761767] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5741.188297] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5747.891771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5748.227442] Lustre: 181205:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5754.062314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5755.387647] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5764.833750] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5774.819424] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5774.822328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5774.838591] Lustre: Skipped 3 previous similar messages [ 5774.842663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5774.842671] Lustre: Skipped 6 previous similar messages [ 5778.932068] Lustre: server umount lustre-OST0000 complete [ 5789.556852] Lustre: server umount lustre-MDT0000 complete [ 5790.181747] LustreError: 179237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5790.204518] LustreError: 179237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5793.171557] LustreError: 181204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798733 with bad export cookie 141312532833393574 [ 5793.179123] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5793.194716] LustreError: 181204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5793.469851] Lustre: server umount lustre-MDT0001 complete [ 5803.508678] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 5807.034207] Key type lgssc unregistered [ 5807.359917] LNet: 183065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5807.382726] LNetError: 183065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5807.409046] LNet: Removed LNI 192.168.204.136@tcp [ 5808.337269] Key type .llcrypt unregistered [ 5808.341346] Key type ._llcrypt unregistered [ 5837.873264] Key type ._llcrypt registered [ 5837.876182] Key type .llcrypt registered [ 5838.034771] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 5839.128679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5839.204782] alg: No test for adler32 (adler32-zlib) [ 5840.593519] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 5840.849053] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 5842.512486] Key type lgssc registered [ 5843.707560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5852.885188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5852.905713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5854.782359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5860.371475] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5864.929568] LustreError: 184483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5864.941939] LustreError: 184483:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5869.642652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5869.972720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5874.540599] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5882.132229] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5887.836125] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5894.965641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5895.674223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5899.775437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5901.685386] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5908.767838] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5930.468566] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5930.481814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5930.483145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5930.504083] Lustre: Skipped 1 previous similar message [ 5930.523033] Lustre: Skipped 1 previous similar message [ 5935.585780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5935.591256] Lustre: Skipped 1 previous similar message [ 5936.626708] Lustre: server umount lustre-OST0000 complete [ 5940.705568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5940.708507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5940.712762] Lustre: Skipped 1 previous similar message [ 5945.832752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5945.844260] Lustre: Skipped 2 previous similar messages [ 5946.774258] Lustre: server umount lustre-MDT0000 complete [ 5950.953530] LustreError: 184479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5950.979204] LustreError: 184479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5951.298866] LustreError: 184465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798892 with bad export cookie 3105896793004908671 [ 5951.303370] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5951.347096] LustreError: 184465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5952.125502] Lustre: server umount lustre-MDT0001 complete [ 5970.092421] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 5973.686844] Key type lgssc unregistered [ 5974.078575] LNet: 188060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5974.083888] LNetError: 188060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5974.095168] LNet: Removed LNI 192.168.204.136@tcp [ 5975.016209] Key type .llcrypt unregistered [ 5975.018311] Key type ._llcrypt unregistered [ 6003.699049] Key type ._llcrypt registered [ 6003.701301] Key type .llcrypt registered [ 6003.798239] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 6005.142023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6005.185350] alg: No test for adler32 (adler32-zlib) [ 6006.171237] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 6006.524567] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 6008.249253] Key type lgssc registered [ 6009.590585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6017.557042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6017.643368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6019.374490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6024.486774] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6029.800133] LustreError: 189482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6029.841640] LustreError: 189482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6034.916084] LustreError: 189483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6035.001523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6035.332872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6040.685358] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6049.397602] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6055.895989] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6063.320267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6063.727749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6064.770262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6070.125256] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6078.717269] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6090.318695] Lustre: lustre-OST0000: Not available for connect from 192.168.204.36@tcp (stopping) [ 6090.743298] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6090.763807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6091.767874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6091.768953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6091.776807] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6091.804796] Lustre: Skipped 2 previous similar messages [ 6093.664722] Lustre: server umount lustre-OST0000 complete [ 6101.529750] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffffa0d706dea800.ost_server_uuid 50 [ 6103.336585] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6117.349953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6117.367365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6117.367490] Lustre: Skipped 1 previous similar message [ 6117.385971] Lustre: Skipped 1 previous similar message [ 6122.469323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6122.477171] Lustre: Skipped 1 previous similar message [ 6123.596161] Lustre: server umount lustre-MDT0000 complete [ 6127.249450] LustreError: 189462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799068 with bad export cookie 17818669045565488480 [ 6127.263130] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6127.547569] Lustre: server umount lustre-MDT0001 complete [ 6136.901433] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 22:51:15 (1787799075) [ 6144.611883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6145.145896] LustreError: 193400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6145.216160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6149.587405] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6150.628250] LustreError: 193401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6154.727232] LustreError: 193400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6158.851474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6159.575520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6166.350968] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6178.365896] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6185.284817] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6192.493874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6192.840243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6200.164951] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6208.012635] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6214.340140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6216.354017] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6216.710186] Lustre: lustre-OST0001: new disk, initializing [ 6216.712992] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6216.720240] Lustre: 196245:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6216.833562] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6223.455677] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6223.938410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6223.952891] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6224.085154] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6233.234923] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6239.337394] Lustre: lustre-OST0000: Not available for connect from 192.168.204.36@tcp (stopping) [ 6239.722723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6242.512527] Lustre: server umount lustre-OST0000 complete [ 6244.403080] LustreError: 196140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6248.924391] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6249.141412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6253.507608] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6253.796477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6265.323649] LustreError: 195222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6265.360324] LustreError: 195222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6270.433474] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6270.441509] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6270.447718] LustreError: Skipped 1 previous similar message [ 6270.452753] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6270.470782] Lustre: Skipped 2 previous similar messages [ 6270.472892] Lustre: Skipped 2 previous similar messages [ 6273.412585] Lustre: server umount lustre-OST0001 complete [ 6277.089476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6280.696226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6280.715574] Lustre: Skipped 1 previous similar message [ 6283.224233] Lustre: server umount lustre-MDT0000 complete [ 6286.964376] LustreError: 193381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799227 with bad export cookie 17818669045565489551 [ 6286.971776] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6286.979531] LustreError: 193381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6287.328167] Lustre: server umount lustre-MDT0001 complete [ 6308.312179] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 22:54:07 (1787799247) [ 6313.912547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6314.137724] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6314.176549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6314.400858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6318.549610] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6327.445917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6327.611394] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6327.665290] Lustre: 199690:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6327.782118] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6327.795342] Lustre: Skipped 4 previous similar messages [ 6332.432392] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6341.359929] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6347.456280] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6354.786997] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6354.968765] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6360.923684] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6361.699601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6368.545306] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6375.926682] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6376.033751] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6383.967434] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6394.356471] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6397.923487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6397.934521] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6397.969485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6404.095984] Lustre: server umount lustre-OST0000 complete [ 6405.090488] LustreError: 201536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6405.108421] LustreError: 201536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6425.579518] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6427.649608] Lustre: server umount lustre-OST0001 complete [ 6435.297653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6435.309982] Lustre: Skipped 3 previous similar messages [ 6439.443687] Lustre: server umount lustre-MDT0000 complete [ 6443.703983] LustreError: 199682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799384 with bad export cookie 17818669045565490958 [ 6443.718085] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6444.103918] Lustre: server umount lustre-MDT0001 complete [ 6473.850239] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 22:56:51 (1787799411) [ 6493.843792] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 6505.329638] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 6538.976137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6539.243493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6539.299899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6539.374966] Lustre: lustre-MDT0000: new disk, initializing [ 6539.432151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6539.436797] Lustre: Skipped 3 previous similar messages [ 6539.451851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6544.392658] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6554.769169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6554.951028] Lustre: 207790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6555.036714] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6555.046030] Lustre: Skipped 1 previous similar message [ 6555.127966] Lustre: lustre-MDT0001: new disk, initializing [ 6555.253475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6555.262778] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6560.387424] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6570.642679] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6576.879574] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6585.627757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6586.121570] Lustre: lustre-OST0000: new disk, initializing [ 6586.125972] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6586.147672] Lustre: 209760:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6593.085277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6593.095560] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6593.192097] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6594.187737] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6605.606828] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6611.489486] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6611.795626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6617.561119] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6617.787326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6624.239879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6624.262733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6624.266087] Lustre: Skipped 2 previous similar messages [ 6624.289713] Lustre: Skipped 8 previous similar messages [ 6626.407649] Lustre: server umount lustre-OST0000 complete [ 6631.905201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6631.914533] LustreError: Skipped 1 previous similar message [ 6645.216267] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6645.472122] Lustre: server umount lustre-MDT0000 complete [ 6649.043338] LustreError: 208584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799589 with bad export cookie 17818669045565492519 [ 6649.048105] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6649.051501] LustreError: 208584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6649.450752] Lustre: server umount lustre-MDT0001 complete [ 6658.243434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6658.780704] LustreError: 211981:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6658.807707] LustreError: 211981:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6664.193860] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6673.723759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6674.325941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6674.339698] Lustre: Skipped 3 previous similar messages [ 6679.807253] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6687.940814] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6693.729897] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6701.007245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6707.933649] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6716.076933] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6721.581965] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6721.871408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6726.386881] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6726.569933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6729.187601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6734.654576] Lustre: server umount lustre-OST0000 complete [ 6740.767880] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6745.400850] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6775.215941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6782.262902] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6784.744034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6784.770600] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6784.790162] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6784.806546] Lustre: Skipped 1 previous similar message [ 6790.694196] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6792.476974] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6800.354091] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6800.355929] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6800.381941] LustreError: Skipped 2 previous similar messages [ 6800.383339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6800.383356] Lustre: Skipped 11 previous similar messages [ 6800.419105] Lustre: Skipped 5 previous similar messages [ 6803.801582] Lustre: server umount lustre-OST0000 complete [ 6814.895955] Lustre: server umount lustre-MDT0000 complete [ 6815.713979] LustreError: 211981:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6815.746592] LustreError: 211981:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6819.844361] LustreError: 211961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799760 with bad export cookie 17818669045565493401 [ 6819.852790] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6819.865226] LustreError: 211961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6820.264129] Lustre: server umount lustre-MDT0001 complete [ 6829.992493] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 23:02:48 (1787799768) [ 6837.204907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6842.953354] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6852.984527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6858.721065] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6866.189883] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6872.063205] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6879.256237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6886.289118] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6894.058699] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6899.278250] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6899.560576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6905.327920] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6905.614753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6913.795587] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6914.148214] Lustre: lustre-OST0001: new disk, initializing [ 6914.154706] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6914.161348] Lustre: 221334:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6921.786613] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6923.873342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6923.889040] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6923.965105] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6931.869878] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6937.128984] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6937.503795] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6941.887483] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6942.207089] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6949.860988] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6949.877934] LustreError: Skipped 1 previous similar message [ 6959.073743] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6959.221227] Lustre: server umount lustre-OST0001 complete [ 6967.593501] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6967.845756] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6972.851682] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6973.167125] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7002.511828] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7002.703114] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7002.713977] Lustre: Skipped 6 previous similar messages [ 7007.728687] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7007.756241] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7007.757703] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7007.781641] Lustre: Skipped 1 previous similar message [ 7009.971989] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7017.708631] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7019.256458] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7027.519335] Lustre: server umount lustre-OST0001 complete [ 7050.606530] Lustre: server umount lustre-OST0000 complete [ 7058.405370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7058.415972] Lustre: Skipped 8 previous similar messages [ 7058.420358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7058.423012] Lustre: Skipped 13 previous similar messages [ 7061.276017] Lustre: server umount lustre-MDT0000 complete [ 7064.879980] LustreError: 217942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800005 with bad export cookie 17818669045565494367 [ 7064.884829] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7064.888958] LustreError: 217942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7079.393548] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7079.880275] Lustre: server umount lustre-MDT0001 complete [ 7105.801413] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 23:07:24 (1787800044) [ 7113.080066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7113.450142] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7113.511338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7118.831971] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7127.348591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7127.415497] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7127.441099] Lustre: 226581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7131.970373] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7140.235996] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7146.126112] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7153.156350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7153.352885] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7159.794942] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7167.544375] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7176.070105] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7176.182723] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7184.554133] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7193.483464] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7195.202352] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7200.832445] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7201.101478] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7206.621101] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7206.886260] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7213.003327] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa0d706192800.ost_server_uuid 50 [ 7215.298322] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa0d706192800.ost_server_uuid in IDLE state after 0 sec [ 7216.692159] Lustre: Permanently deactivating lustre-OST0001 [ 7216.736359] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7221.396501] Lustre: Permanently reactivating lustre-OST0001 [ 7221.435544] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7221.437869] Lustre: Skipped 2 previous similar messages [ 7230.466212] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7230.480390] LustreError: Skipped 6 previous similar messages [ 7235.541203] Lustre: server umount lustre-OST0001 complete [ 7235.575613] LustreError: 228428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7235.601992] LustreError: 228428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 41 previous similar messages [ 7253.984266] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7254.225080] Lustre: server umount lustre-OST0000 complete [ 7265.684959] Lustre: server umount lustre-MDT0000 complete [ 7270.166832] LustreError: 226575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800210 with bad export cookie 17818669045565495655 [ 7270.167300] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7270.185913] LustreError: 226575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7270.644349] Lustre: server umount lustre-MDT0001 complete [ 7295.159689] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 23:10:33 (1787800233) [ 7303.335131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7303.724312] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7303.769489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7303.787741] Lustre: Skipped 2 previous similar messages [ 7309.099242] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7317.482728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7317.573315] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7317.615055] Lustre: 233282:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7322.010172] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7329.787862] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7338.639706] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7348.654293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7348.855553] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7348.880811] Lustre: Permanently deactivating lustre-OST0000 [ 7348.895557] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7355.165441] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7358.025858] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7363.665367] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7371.972124] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7372.039523] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7380.446443] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7381.829967] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7388.565107] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7390.739901] Lustre: Permanently reactivating lustre-OST0000 [ 7390.760397] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7390.766601] Lustre: Skipped 2 previous similar messages [ 7400.500669] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0001-mdtlov_UUID (at 0@lo) in 43 seconds. I think it's dead, and I am evicting it. exp 000000009b7dcecc, cur 1787800341 deadline 1787800328 last 1787800298 [ 7401.368320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7419.360131] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7419.577548] Lustre: server umount lustre-OST0001 complete [ 7429.847853] Lustre: server umount lustre-OST0000 complete [ 7440.142771] Lustre: server umount lustre-MDT0000 complete [ 7443.998543] LustreError: 233274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800384 with bad export cookie 17818669045565497328 [ 7444.004131] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7444.011418] LustreError: 233274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7444.443762] Lustre: server umount lustre-MDT0001 complete [ 7455.293801] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 7458.593186] Key type lgssc unregistered [ 7458.934668] LNet: 238173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7458.941393] LNetError: 238173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7458.961822] LNet: Removed LNI 192.168.204.136@tcp [ 7459.887252] Key type .llcrypt unregistered [ 7459.889927] Key type ._llcrypt unregistered [ 7480.832684] Key type ._llcrypt registered [ 7480.834821] Key type .llcrypt registered [ 7480.959899] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 7482.056772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7482.079968] alg: No test for adler32 (adler32-zlib) [ 7483.157892] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 7483.454045] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 7485.112156] Key type lgssc registered [ 7486.309683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7495.829501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7495.853784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7497.244131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7501.331811] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7506.404743] LustreError: 239620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7506.414410] LustreError: 239620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7508.560782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7508.686872] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7508.723717] Lustre: 239608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7508.735555] Lustre: Permanently deactivating lustre-MDT0001 [ 7508.756364] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7509.064861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7512.266349] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7515.152968] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7519.159252] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7529.711558] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7539.366429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7539.864675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7541.933675] LustreError: 241464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7541.969163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7546.837897] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7547.374417] LustreError: 241576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7552.493683] LustreError: 241462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7555.031518] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7558.642421] LustreError: 241462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7558.659449] LustreError: 241462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7561.744666] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7562.048304] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7567.342990] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7567.371502] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7569.171884] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7577.631223] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7581.916726] Lustre: Permanently reactivating lustre-MDT0001 [ 7581.929375] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7581.934483] Lustre: Skipped 2 previous similar messages [ 7587.401360] Lustre: lustre-MDT0001: Client 4ba9c683-6573-422c-9d07-1dc52a11adf5 (at 192.168.204.36@tcp) reconnecting [ 7590.595312] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7590.611631] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7590.632673] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7590.640267] LustreError: 243315:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7590.645836] LustreError: 243315:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7590.648700] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7590.655174] LustreError: 243315:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.136@tcp: cfg command failed: rc = -108 [ 7590.665191] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7590.665191] [ 7590.670899] LustreError: 239612:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7592.856969] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7594.281738] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7599.853735] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7600.128637] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7604.874239] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7605.135063] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7607.793760] Lustre: Permanently deactivating lustre-MDT0001 [ 7607.813509] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7607.819679] Lustre: Skipped 2 previous similar messages [ 7618.174994] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7630.337989] Lustre: server umount lustre-MDT0000 complete [ 7631.842583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7631.844363] LustreError: 241575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7631.866932] Lustre: Skipped 1 previous similar message [ 7633.449600] LustreError: 240359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800574 with bad export cookie 3554749925681023192 [ 7633.454765] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7633.466639] LustreError: 240359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7633.769288] Lustre: server umount lustre-MDT0001 complete [ 7637.019834] Lustre: server umount lustre-OST0000 complete [ 7649.017815] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:16:27 (1787800587) [ 7659.110978] Lustre: server umount lustre-OST0001 complete [ 7665.462366] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 7674.631921] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 7714.684437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7714.940923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7714.956795] Lustre: Skipped 2 previous similar messages [ 7714.976533] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7715.045615] Lustre: lustre-MDT0000: new disk, initializing [ 7715.155639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7715.194647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7720.188729] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7731.639574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7731.802971] Lustre: 247931:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7731.874058] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7731.878750] Lustre: Skipped 1 previous similar message [ 7731.985888] Lustre: lustre-MDT0001: new disk, initializing [ 7732.130896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7732.196985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7732.221248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7737.328290] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7748.240232] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7755.266734] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7763.019625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7763.316853] Lustre: lustre-OST0000: new disk, initializing [ 7763.320750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7763.326700] Lustre: 249901:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7763.457198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7768.679834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7768.687245] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7768.807816] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7770.307911] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7781.456725] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7784.141636] LustreError: 249349:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7793.525146] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7793.730838] Lustre: lustre-OST0001: new disk, initializing [ 7793.733891] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7793.738105] Lustre: 251015:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7794.221680] LustreError: 249349:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7794.267849] LustreError: 247939:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7801.316788] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7802.509199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7802.527298] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7802.672726] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7804.288783] LustreError: 247939:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7812.876724] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7814.726654] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7818.222234] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7818.227910] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7818.244411] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7818.271931] Lustre: Skipped 2 previous similar messages [ 7823.344374] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7823.357953] Lustre: Skipped 2 previous similar messages [ 7828.459509] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7828.465266] Lustre: Skipped 1 previous similar message [ 7831.008181] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7831.086692] Lustre: server umount lustre-OST0001 complete [ 7833.579446] LustreError: 249884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7833.603459] LustreError: 249884:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7848.935153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7848.953562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7848.958304] Lustre: Skipped 1 previous similar message [ 7848.967742] Lustre: Skipped 1 previous similar message [ 7853.614780] Lustre: server umount lustre-OST0000 complete [ 7859.174821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7859.198075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7859.210914] Lustre: Skipped 1 previous similar message [ 7863.839350] Lustre: server umount lustre-MDT0000 complete [ 7867.655811] LustreError: 247925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800808 with bad export cookie 3554749925681025789 [ 7867.656727] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7867.663347] LustreError: 247925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7874.170402] Lustre: server umount lustre-MDT0001 complete [ 7884.793141] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 7887.871273] Key type lgssc unregistered [ 7888.164634] LNet: 253053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7888.174178] LNetError: 253053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7888.187771] LNet: Removed LNI 192.168.204.136@tcp [ 7889.008639] Key type .llcrypt unregistered [ 7889.011018] Key type ._llcrypt unregistered [ 7929.569315] Key type ._llcrypt registered [ 7929.577671] Key type .llcrypt registered [ 7929.695069] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 7930.995663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7931.033839] alg: No test for adler32 (adler32-zlib) [ 7932.037607] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 7932.351096] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 7934.096226] Key type lgssc registered [ 7935.314607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7943.608714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7943.621954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7944.795937] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7944.818918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7944.937815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7949.871927] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7959.358380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7959.577802] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7959.665778] Lustre: 255271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7960.130140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7967.028700] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7976.706870] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7983.960347] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7992.348404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7992.673416] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7993.114994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8000.827421] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8003.392974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8010.964807] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8070.112760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8070.122654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8070.123341] LustreError: Skipped 1 previous similar message [ 8070.138507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8072.956901] Lustre: server umount lustre-OST0000 complete [ 8077.156431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8089.530383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8090.044839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8091.057040] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8091.081680] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8091.086543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8096.422948] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8104.694862] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8147.425520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8147.427933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8147.444606] LustreError: Skipped 1 previous similar message [ 8147.445627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8147.445632] Lustre: Skipped 1 previous similar message [ 8147.497923] Lustre: Skipped 2 previous similar messages [ 8150.693423] Lustre: server umount lustre-OST0000 complete [ 8157.669464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8157.678925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8157.685772] Lustre: Skipped 1 previous similar message [ 8157.701523] Lustre: Skipped 2 previous similar messages [ 8161.167272] Lustre: server umount lustre-MDT0000 complete [ 8165.153837] LustreError: 255263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801105 with bad export cookie 1066039717947297406 [ 8165.159075] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8165.166802] LustreError: 255263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8165.579202] Lustre: server umount lustre-MDT0001 complete [ 8176.103193] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 8179.263134] Key type lgssc unregistered [ 8179.543334] LNet: 260700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8179.554445] LNetError: 260700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8179.592271] LNet: Removed LNI 192.168.204.136@tcp [ 8180.570288] Key type .llcrypt unregistered [ 8180.577102] Key type ._llcrypt unregistered [ 8204.141733] Key type ._llcrypt registered [ 8204.145085] Key type .llcrypt registered [ 8204.248642] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 8205.355474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8205.377299] alg: No test for adler32 (adler32-zlib) [ 8206.449583] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 8206.676263] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 8208.320727] Key type lgssc registered [ 8209.472398] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8217.941546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8217.963031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8219.381029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8223.885359] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8229.857509] LustreError: 262111:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8229.889904] LustreError: 262111:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8232.732158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8233.034555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8237.307828] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8244.349283] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8249.380151] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8256.070383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8256.550946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8262.748402] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8264.766903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8271.146290] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8298.467895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8298.474736] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8298.489898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8300.523662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8300.549720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8302.783873] Lustre: server umount lustre-OST0000 complete [ 8309.731274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8309.742580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8309.757357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8309.761356] Lustre: Skipped 1 previous similar message [ 8313.910070] Lustre: server umount lustre-MDT0000 complete [ 8315.873265] LustreError: 263402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8315.901926] LustreError: 263402:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8318.217996] LustreError: 265666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801258 with bad export cookie 15624396368811781728 [ 8318.225899] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8318.241367] LustreError: 265666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8318.757960] Lustre: server umount lustre-MDT0001 complete [ 8329.252326] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 8332.334710] Key type lgssc unregistered [ 8332.724860] LNet: 266420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8332.740781] LNetError: 266420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8332.767618] LNet: Removed LNI 192.168.204.136@tcp [ 8333.736250] Key type .llcrypt unregistered [ 8333.738408] Key type ._llcrypt unregistered [ 8349.871734] Key type ._llcrypt registered [ 8349.873370] Key type .llcrypt registered [ 8350.001751] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 8351.077370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8351.154942] alg: No test for adler32 (adler32-zlib) [ 8352.255778] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 8352.645711] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 8354.368175] Key type lgssc registered [ 8355.634850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8364.069325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8364.118954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8365.619436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8369.888804] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8375.780843] LustreError: 267643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8375.806095] LustreError: 267643:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8378.939711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8379.493208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8384.355764] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8391.853995] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8397.187995] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8404.403879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8404.885420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8411.136276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8411.607229] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8420.037600] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8431.590427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8431.593212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8431.595198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8431.598057] Lustre: Skipped 1 previous similar message [ 8431.610022] LustreError: Skipped 1 previous similar message [ 8436.687582] Lustre: server umount lustre-OST0000 complete [ 8440.801487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8440.814436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8440.827432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8440.829778] Lustre: Skipped 1 previous similar message [ 8441.826237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8441.851188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8441.862148] Lustre: Skipped 1 previous similar message [ 8446.244380] Lustre: server umount lustre-MDT0000 complete [ 8446.952700] LustreError: 267643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8446.997502] LustreError: 267643:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8450.294436] LustreError: 269488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801391 with bad export cookie 16041788520189111050 [ 8450.305568] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8450.313514] LustreError: 269488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8450.813852] Lustre: server umount lustre-MDT0001 complete [ 8464.777706] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 8467.926088] Key type lgssc unregistered [ 8468.232903] LNet: 271323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8468.241929] LNetError: 271323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8468.265683] LNet: Removed LNI 192.168.204.136@tcp [ 8469.165953] Key type .llcrypt unregistered [ 8469.170109] Key type ._llcrypt unregistered [ 8495.604148] Key type ._llcrypt registered [ 8495.605432] Key type .llcrypt registered [ 8495.709890] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 8496.561122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8496.597061] alg: No test for adler32 (adler32-zlib) [ 8497.652487] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 8497.887907] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 8499.524498] Key type lgssc registered [ 8500.426155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8509.142666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8509.164979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8510.988557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8515.778370] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8521.189263] LustreError: 272741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8521.208942] LustreError: 272741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8524.453243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8524.691023] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8529.481597] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8537.516675] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8544.361227] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8552.541026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8552.857125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8555.961537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8558.854994] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8567.759344] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8594.920591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8594.925602] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8594.930150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8594.985734] Lustre: Skipped 1 previous similar message [ 8599.374622] Lustre: server umount lustre-OST0000 complete [ 8605.153287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8605.163291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8605.167662] Lustre: Skipped 2 previous similar messages [ 8610.041484] Lustre: server umount lustre-MDT0000 complete [ 8610.274482] LustreError: 273984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8610.289113] LustreError: 273984:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8613.423405] LustreError: 274588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801554 with bad export cookie 1637251226253245674 [ 8613.428108] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8613.437565] LustreError: 274588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8613.766778] Lustre: server umount lustre-MDT0001 complete [ 8627.496877] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 8630.802336] Key type lgssc unregistered [ 8631.111547] LNet: 277055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8631.124108] LNetError: 277055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8631.149289] LNet: Removed LNI 192.168.204.136@tcp [ 8632.158484] Key type .llcrypt unregistered [ 8632.160296] Key type ._llcrypt unregistered [ 8650.899271] Key type ._llcrypt registered [ 8650.902520] Key type .llcrypt registered [ 8651.039282] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 8652.168994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8652.376294] alg: No test for adler32 (adler32-zlib) [ 8653.652628] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 8653.946972] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 8655.648181] Key type lgssc registered [ 8656.892683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8665.149531] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8665.176954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8666.745675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8671.109641] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8676.345173] LustreError: 278292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8676.361909] LustreError: 278292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8678.735369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8679.108990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8683.692293] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8691.066907] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8696.253558] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8702.028112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8702.365213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8708.172315] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8711.610151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8716.521417] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8729.058472] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8729.067354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8729.081230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8732.138932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8732.144777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8732.149787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8733.553182] Lustre: server umount lustre-OST0000 complete [ 8740.835821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8740.850625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8740.865597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8740.871344] Lustre: Skipped 1 previous similar message [ 8744.380546] Lustre: server umount lustre-MDT0000 complete [ 8747.505464] LustreError: 278293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8747.526909] LustreError: 278293:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8748.350081] LustreError: 278258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801689 with bad export cookie 12919020162970572764 [ 8748.358227] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8748.362073] LustreError: 278258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8749.013498] Lustre: server umount lustre-MDT0001 complete [ 8761.217396] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 8764.228750] Key type lgssc unregistered [ 8764.582160] LNet: 281996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8764.589608] LNetError: 281996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8764.608335] LNet: Removed LNI 192.168.204.136@tcp [ 8765.461676] Key type .llcrypt unregistered [ 8765.464719] Key type ._llcrypt unregistered [ 8777.898180] Key type ._llcrypt registered [ 8777.902565] Key type .llcrypt registered [ 8777.996373] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8795.419849] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 8811.470379] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 8812.729932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8812.834394] alg: No test for adler32 (adler32-zlib) [ 8814.006860] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 8814.282702] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 8816.000183] Key type lgssc registered [ 8817.397567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8854.735723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8854.788286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8856.087297] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8856.139304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8856.260749] Lustre: lustre-MDT0000: new disk, initializing [ 8856.388283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8856.415832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8861.188100] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8872.305677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8872.472288] Lustre: 285548:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8872.554358] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8872.563386] Lustre: Skipped 1 previous similar message [ 8872.676347] Lustre: lustre-MDT0001: new disk, initializing [ 8872.807067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8872.854439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8872.880452] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8877.812316] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8887.902293] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8894.156967] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8901.818125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8902.210578] Lustre: lustre-OST0000: new disk, initializing [ 8902.216885] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8902.246652] Lustre: 287516:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8902.457573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8907.854470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8907.875458] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8908.005147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8909.119756] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8918.802772] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8924.440921] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8924.668515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8930.018466] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8930.320551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8933.858248] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8933.861584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8933.869981] LustreError: Skipped 1 previous similar message [ 8933.873288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8933.906286] Lustre: Skipped 1 previous similar message [ 8938.632555] Lustre: server umount lustre-OST0000 complete [ 8944.101265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8944.134216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8944.140323] Lustre: Skipped 1 previous similar message [ 8949.218510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8949.232360] Lustre: Skipped 2 previous similar messages [ 8949.326248] Lustre: server umount lustre-MDT0000 complete [ 8953.027208] LustreError: 285538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801893 with bad export cookie 2824650905934535275 [ 8953.033370] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8953.035446] LustreError: 285538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8953.296976] Lustre: server umount lustre-MDT0001 complete [ 8962.106381] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 23:38:20 (1787801900) [ 8968.330271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8968.844146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8973.526560] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8974.312333] LustreError: 289929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8974.341791] LustreError: 289929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8978.421475] LustreError: 289928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8982.446518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8982.910405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8987.507757] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8995.172432] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9000.822325] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9008.213526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9008.645935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9015.020499] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9023.239445] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9034.226693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9034.236996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9034.238343] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9034.258583] Lustre: Skipped 2 previous similar messages [ 9034.289121] Lustre: Skipped 1 previous similar message [ 9039.228058] Lustre: server umount lustre-OST0000 complete [ 9044.450828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9044.453769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9044.481406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9044.491708] Lustre: Skipped 1 previous similar message [ 9050.025646] Lustre: server umount lustre-MDT0000 complete [ 9054.091755] LustreError: 289909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801994 with bad export cookie 2824650905934536150 [ 9054.098591] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9054.119663] LustreError: 289909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9054.458466] Lustre: server umount lustre-MDT0001 complete [ 9066.950851] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 9070.353889] Key type lgssc unregistered [ 9070.771191] LNet: 293465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9070.786870] LNetError: 293465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9070.816982] LNet: Removed LNI 192.168.204.136@tcp [ 9071.764994] Key type .llcrypt unregistered [ 9071.772873] Key type ._llcrypt unregistered [ 9105.137139] Key type ._llcrypt registered [ 9105.141703] Key type .llcrypt registered [ 9105.331349] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 9122.722341] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 9124.546883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9124.669643] alg: No test for adler32 (adler32-zlib) [ 9125.921565] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 9126.228975] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 9127.969092] Key type lgssc registered [ 9129.682906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9189.631451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9189.664698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9190.936543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9190.998848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9191.135089] Lustre: lustre-MDT0000: new disk, initializing [ 9191.217390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9191.265464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9197.325479] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9210.028305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9210.391101] Lustre: 297298:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9210.556595] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9210.569082] Lustre: Skipped 1 previous similar message [ 9210.691149] Lustre: lustre-MDT0001: new disk, initializing [ 9210.916833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9210.998471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9211.017642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9217.086851] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9227.858519] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9235.476972] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9243.358337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9243.644248] Lustre: lustre-OST2710: new disk, initializing [ 9243.647528] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9243.654431] Lustre: 299265:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9243.713194] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9249.601452] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9252.978050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9253.005727] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9253.132860] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9261.685807] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9270.976376] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9271.274926] Lustre: lustre-OST03e8: new disk, initializing [ 9271.279539] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9271.284941] Lustre: 300328:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9271.409883] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9278.645526] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9279.184237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9279.193447] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9279.372973] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9288.825851] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9295.864847] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9296.064620] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9301.491119] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9301.706889] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9307.140798] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9307.386834] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9312.242145] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9312.436158] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9318.370284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9318.376104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9318.402194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9319.689639] Lustre: server umount lustre-MDT0000 complete [ 9320.426024] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9323.274101] LustreError: 298091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802264 with bad export cookie 4589625157590595882 [ 9323.280814] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9323.289569] LustreError: 298091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9324.000193] Lustre: server umount lustre-MDT0001 complete [ 9338.667539] Lustre: server umount lustre-OST2710 complete [ 9341.667632] Lustre: 295092:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787802266/real 1787802266] req@000000004aaf5e58 x1874646339229312/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787802282 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9341.692196] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9341.702375] Lustre: Skipped 2 previous similar messages [ 9342.614577] Lustre: server umount lustre-OST03e8 complete [ 9361.300804] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 9372.535780] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 9414.003649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9414.421286] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9414.440425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9414.534175] Lustre: lustre-MDT0000: new disk, initializing [ 9414.587438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9414.608331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9419.348562] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9430.786153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9430.924763] Lustre: 305576:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9430.974924] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9430.982358] Lustre: Skipped 1 previous similar message [ 9431.140628] Lustre: lustre-MDT0001: new disk, initializing [ 9431.259279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9431.309603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9431.319771] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9435.896202] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9445.772298] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9451.462355] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9459.305340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9459.676815] Lustre: lustre-OST0000: new disk, initializing [ 9459.680890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9459.693585] Lustre: 307545:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9459.821116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9466.076114] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9468.462408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9468.475312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9468.602381] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9475.163597] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9479.765601] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9480.000810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9484.406436] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9484.622331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9488.865866] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9488.876747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9488.895736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9492.594230] Lustre: server umount lustre-OST0000 complete [ 9498.095807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9498.109792] LustreError: Skipped 1 previous similar message [ 9498.122192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9498.141031] Lustre: Skipped 1 previous similar message [ 9498.165222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9498.177221] Lustre: Skipped 2 previous similar messages [ 9503.821614] Lustre: server umount lustre-MDT0000 complete [ 9504.738560] LustreError: 305589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9504.768131] LustreError: 305589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9508.648858] LustreError: 305569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802449 with bad export cookie 4589625157590597569 [ 9508.649337] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9508.689942] LustreError: 305569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9509.187500] Lustre: server umount lustre-MDT0001 complete [ 9519.445481] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 23:47:38 (1787802458) [ 9521.368388] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9523.421796] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 23:47:41 (1787802461) [ 9527.256188] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 9554.622841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9554.859886] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9554.881499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9555.042570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9560.380733] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9569.470055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9569.583580] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9569.616809] Lustre: 311405:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9574.959494] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9596.788986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9596.976687] Lustre: Denying initial registration attempt from nid 192.168.204.136@tcp, specified as failover [ 9596.982873] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9596.990686] LustreError: 312858:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9596.997617] LustreError: 312858:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9597.002260] LustreError: 312858:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9597.070562] Lustre: server umount lustre-OST0000 complete [ 9597.075137] LustreError: 312858:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9604.581646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9604.594865] Lustre: Skipped 1 previous similar message [ 9604.600242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9604.611083] Lustre: Skipped 3 previous similar messages [ 9609.698830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9609.707607] Lustre: Skipped 1 previous similar message [ 9610.904861] Lustre: server umount lustre-MDT0000 complete [ 9614.823740] LustreError: 311412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9614.845608] LustreError: 311412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9615.151502] LustreError: 311399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802555 with bad export cookie 4589625157590598451 [ 9615.167825] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9615.779987] Lustre: server umount lustre-MDT0001 complete [ 9626.292947] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 9629.044583] Key type lgssc unregistered [ 9629.360620] LNet: 313980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9629.370173] LNetError: 313980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9629.386821] LNet: Removed LNI 192.168.204.136@tcp [ 9630.278127] Key type .llcrypt unregistered [ 9630.280873] Key type ._llcrypt unregistered [ 9640.551955] Key type ._llcrypt registered [ 9640.554746] Key type .llcrypt registered [ 9640.641426] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 9641.528046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9641.635994] alg: No test for adler32 (adler32-zlib) [ 9642.656334] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 9642.879564] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 9644.536452] Key type lgssc registered [ 9645.787101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9693.038565] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 9704.758936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9704.795692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9706.139214] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9706.177919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9706.403478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9711.711624] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9720.791040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9720.927383] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9720.961210] Lustre: 316744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9721.274312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9726.415471] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9735.234985] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9741.304283] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9761.526647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9761.792451] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9762.055932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9768.646942] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9776.378295] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9780.718842] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9780.728252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9780.746385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9784.537924] Lustre: server umount lustre-OST0000 complete [ 9790.946036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9790.951952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9790.962264] Lustre: Skipped 2 previous similar messages [ 9790.970051] Lustre: Skipped 2 previous similar messages [ 9796.749523] Lustre: server umount lustre-MDT0000 complete [ 9800.557947] LustreError: 317488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802741 with bad export cookie 8197542735006559455 [ 9800.561596] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9800.565660] LustreError: 317488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9801.072639] Lustre: server umount lustre-MDT0001 complete [ 9812.667213] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 9815.108466] Key type lgssc unregistered [ 9815.381664] LNet: 320324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9815.388830] LNetError: 320324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9815.406598] LNet: Removed LNI 192.168.204.136@tcp [ 9816.239428] Key type .llcrypt unregistered [ 9816.244417] Key type ._llcrypt unregistered [ 9840.879336] Key type ._llcrypt registered [ 9840.881697] Key type .llcrypt registered [ 9841.047539] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 9842.648280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9842.677820] alg: No test for adler32 (adler32-zlib) [ 9843.892324] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [ 9844.147844] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 9845.840278] Key type lgssc registered [ 9847.250904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9856.497182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9856.536943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9858.385239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9863.855359] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9868.782133] LustreError: 321743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9868.814567] LustreError: 321743:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9873.549888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9873.896347] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9874.105230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9877.881361] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9883.322165] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9888.354610] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9895.531586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9895.851357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9901.632721] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9910.168557] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9919.463938] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9919.466391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9919.486834] Lustre: Skipped 2 previous similar messages [ 9919.498164] Lustre: Skipped 2 previous similar messages [ 9923.148422] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.36@tcp (stopping) [ 9925.538997] Lustre: server umount lustre-MDT0000 complete [ 9928.239142] LustreError: 321738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9928.581162] LustreError: 321724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802869 with bad export cookie 15131129290858530449 [ 9928.588553] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9928.594673] LustreError: 321724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9929.698751] LustreError: 323747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9929.701176] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9929.728852] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9929.731100] Lustre: Skipped 3 previous similar messages [ 9933.363590] LustreError: 321739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9933.741418] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9933.749392] Lustre: Skipped 1 previous similar message [ 9934.987637] Lustre: server umount lustre-MDT0001 complete [ 9940.234361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9948.243515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9953.579301] LustreError: 323573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9953.614817] LustreError: 323573:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9953.733346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9953.825310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9958.161784] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9963.062678] LustreError: 325331:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9963.079991] LustreError: 325331:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9963.952379] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9966.542776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9967.143637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9972.212860] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9972.597672] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9980.619068] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9982.083626] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9987.833140] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9989.395618] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9994.724386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9994.732430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9994.742373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9997.797356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10000.849429] Lustre: server umount lustre-OST0000 complete [10008.044626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10008.057686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10008.068513] Lustre: Skipped 1 previous similar message [10011.596380] Lustre: server umount lustre-MDT0000 complete [10013.155511] LustreError: 325332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10013.168238] LustreError: 325332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10015.216189] LustreError: 322476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802955 with bad export cookie 15131129290858537589 [10015.223312] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10015.232956] LustreError: 322476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10015.551471] Lustre: server umount lustre-MDT0001 complete [10025.083666] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [10027.398307] Key type lgssc unregistered [10027.700850] LNet: 328012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10027.706453] LNetError: 328012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10027.732222] LNet: Removed LNI 192.168.204.136@tcp [10028.546686] Key type .llcrypt unregistered [10028.549604] Key type ._llcrypt unregistered [10052.194123] Key type ._llcrypt registered [10052.196153] Key type .llcrypt registered [10052.285355] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [10053.132586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10053.165293] alg: No test for adler32 (adler32-zlib) [10054.277692] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [10054.562805] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [10056.267065] Key type lgssc registered [10057.592621] Lustre: Echo OBD driver; http://www.lustre.org/ [10067.300503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10067.307462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10068.600352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10073.055892] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10079.205856] LustreError: 329430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10079.235809] LustreError: 329430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10082.281025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10082.739524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10089.198818] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10098.734575] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10106.910863] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10112.485170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10112.494442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10112.496910] Lustre: Skipped 1 previous similar message [10112.504795] Lustre: Skipped 1 previous similar message [10117.622994] LustreError: 329430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10117.828355] Lustre: server umount lustre-MDT0000 complete [10122.060043] LustreError: 329412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803062 with bad export cookie 7370171267673307689 [10122.077546] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10122.868296] Lustre: server umount lustre-MDT0001 complete [10139.018877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10139.448411] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10139.495378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10139.756456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10144.013705] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10153.562815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10153.715580] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10153.800518] Lustre: 332020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10154.138966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10159.137589] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10167.129399] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10172.701609] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10181.814903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10182.118885] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10182.133059] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10182.151272] LustreError: 333907:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10182.158684] LustreError: 333907:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10182.166845] LustreError: 333907:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10182.247411] Lustre: server umount lustre-OST0000 complete [10182.252054] LustreError: 333907:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10195.064159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10195.511400] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10195.786657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10202.511423] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10203.258031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10209.913463] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10218.576483] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10218.679993] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10218.822994] Lustre: lustre-OST0001: new disk, initializing [10218.827884] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10218.837351] Lustre: 335449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10218.958793] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10225.767117] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10226.350621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10226.373722] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10226.500126] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10234.842511] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10242.022670] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10242.027138] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10242.044602] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10242.072589] Lustre: Skipped 1 previous similar message [10242.103284] Lustre: Skipped 1 previous similar message [10244.363322] Lustre: server umount lustre-OST0001 complete [10247.151703] LustreError: 334375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10247.169307] LustreError: 334375:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10249.188093] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10249.201488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10249.221491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10252.259621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10252.261414] LustreError: 334375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10252.268126] Lustre: Skipped 2 previous similar messages [10254.830970] Lustre: server umount lustre-OST0000 complete [10261.479938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10261.488680] LustreError: Skipped 1 previous similar message [10261.495396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10261.513413] Lustre: Skipped 1 previous similar message [10261.521942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10265.118769] Lustre: server umount lustre-MDT0000 complete [10267.617484] LustreError: 332029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10267.627908] LustreError: 332029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10269.238075] LustreError: 332771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803210 with bad export cookie 7370171267673308242 [10269.238642] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10269.254679] LustreError: 332771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10269.601788] Lustre: server umount lustre-MDT0001 complete [10282.047199] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [10285.179752] Key type lgssc unregistered [10285.533314] LNet: 337411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10285.544835] LNetError: 337411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10285.570306] LNet: Removed LNI 192.168.204.136@tcp [10286.473118] Key type .llcrypt unregistered [10286.478847] Key type ._llcrypt unregistered [10364.995083] Key type ._llcrypt registered [10364.997994] Key type .llcrypt registered [10365.101681] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [10381.690725] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [10383.349061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10383.379828] alg: No test for adler32 (adler32-zlib) [10384.720252] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [10385.260403] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [10387.089328] Key type lgssc registered [10388.553877] Lustre: Echo OBD driver; http://www.lustre.org/ [10431.974072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10431.993358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10433.206801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10433.236247] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10433.312965] Lustre: lustre-MDT0000: new disk, initializing [10433.448673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10433.472497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10438.572706] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10449.164411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10449.331723] Lustre: 342361:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10449.394937] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10449.398697] Lustre: Skipped 1 previous similar message [10449.476509] Lustre: lustre-MDT0001: new disk, initializing [10449.584894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10449.641786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10449.651141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10453.889973] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10463.924063] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10469.944830] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10477.995978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10478.252799] Lustre: lustre-OST0000: new disk, initializing [10478.257285] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10478.264938] Lustre: 344329:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10478.371275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10484.653544] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10487.974935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10488.004050] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10488.320479] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10495.188110] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10499.673438] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10499.899564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10504.136188] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10504.563687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10508.770880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10508.771984] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10508.774892] LustreError: Skipped 1 previous similar message [10508.793749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10508.804427] Lustre: Skipped 1 previous similar message [10513.120766] Lustre: server umount lustre-OST0000 complete [10519.015397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10519.017173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10519.046712] Lustre: Skipped 2 previous similar messages [10519.068732] Lustre: Skipped 1 previous similar message [10524.130331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10524.149764] Lustre: Skipped 1 previous similar message [10524.937325] Lustre: server umount lustre-MDT0000 complete [10529.635359] LustreError: 343155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803470 with bad export cookie 16633077837030280232 [10529.640844] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10529.642105] LustreError: 343155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10530.012685] Lustre: server umount lustre-MDT0001 complete [10541.305990] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 00:04:39 (1787803479) [10552.934385] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 00:04:50 (1787803490) [10562.723864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10563.153828] LustreError: 347087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10563.183913] LustreError: 347087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10563.241656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10568.068097] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10568.679415] LustreError: 347088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10572.773986] LustreError: 347087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10577.515122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10577.858894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10584.234474] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10595.447721] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10603.374767] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10610.717739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10611.107559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10617.961363] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10627.245266] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10641.766919] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10644.684916] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10646.662964] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10649.577298] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10651.944803] Lustre: Failing over lustre-MDT0000 [10652.236556] Lustre: server umount lustre-MDT0000 complete [10653.154459] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10653.158232] LustreError: 347815:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10653.162847] Lustre: Skipped 1 previous similar message [10658.277822] LustreError: 347087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10658.302035] LustreError: 347087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10668.513680] LustreError: 347083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10668.514874] Lustre: 340585:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787803593/real 1787803593] req@000000006276b41e x1874647660438400/t0(0) o400->MGC192.168.204.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787803609 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10668.559787] LustreError: 347083:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10668.670950] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10675.843707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10678.639982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10678.704768] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10680.369870] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10683.901038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10683.923907] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10683.990416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10684.471397] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10694.197946] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10696.615889] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10701.220799] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10704.205179] Lustre: Failing over lustre-MDT0000 [10704.354353] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10704.370086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10704.379255] Lustre: Skipped 1 previous similar message [10704.408534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10705.048859] Lustre: server umount lustre-MDT0000 complete [10706.037715] LustreError: 347083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10706.074431] LustreError: 347083:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [10725.857166] Lustre: 340584:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787803650/real 1787803650] req@000000006e828678 x1874647660460032/t0(0) o400->MGC192.168.204.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787803666 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10725.882436] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10726.553857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10736.096837] LustreError: 340583:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ee6c7565 x1874647660464896/t0(0) o250->MGC192.168.204.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10736.472872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10736.537487] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10736.699280] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10741.761371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10741.776359] Lustre: Skipped 2 previous similar messages [10741.829647] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10741.928864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10742.344682] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10752.783537] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10754.400894] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10762.214466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10762.216425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10762.222630] Lustre: Skipped 3 previous similar messages [10762.224859] Lustre: Skipped 3 previous similar messages [10766.885448] Lustre: server umount lustre-MDT0000 complete [10767.333556] LustreError: 347088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10767.377497] LustreError: 347088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 28 previous similar messages [10770.883495] LustreError: 347069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803711 with bad export cookie 16633077837030282948 [10770.885416] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10770.899207] LustreError: 347069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10771.764533] Lustre: server umount lustre-MDT0001 complete [10787.393034] Lustre: server umount lustre-OST0000 complete [10805.382745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10806.030383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10810.834240] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10820.705590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10825.996521] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10833.677755] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10840.304402] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10848.018691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10850.621871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10858.281592] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10867.355438] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10876.385539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10876.397767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10876.408786] LustreError: Skipped 1 previous similar message [10876.439835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10876.439844] Lustre: Skipped 1 previous similar message [10879.760974] Lustre: server umount lustre-OST0000 complete [10896.868137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10896.894884] Lustre: Skipped 6 previous similar messages [10897.888197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10898.049772] Lustre: server umount lustre-MDT0000 complete [10901.995371] LustreError: 353456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10902.015426] LustreError: 353456:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10902.747348] LustreError: 353436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803843 with bad export cookie 16633077837030283683 [10902.752737] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10902.754227] LustreError: 353436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10902.942834] Lustre: server umount lustre-MDT0001 complete [10917.893737] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [10922.066421] Key type lgssc unregistered [10922.519650] LNet: 356994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10922.535362] LNetError: 356994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10922.557417] LNet: Removed LNI 192.168.204.136@tcp [10923.743737] Key type .llcrypt unregistered [10923.745599] Key type ._llcrypt unregistered [10951.045937] Key type ._llcrypt registered [10951.048257] Key type .llcrypt registered [10951.137861] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [10951.860677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10951.868947] alg: No test for adler32 (adler32-zlib) [10952.957283] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [10953.151511] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [10954.800183] Key type lgssc registered [10955.711614] Lustre: Echo OBD driver; http://www.lustre.org/ [10964.382140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10964.397161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10966.210130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10972.381056] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10975.719201] LustreError: 358512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10975.780416] LustreError: 358512:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10980.842721] LustreError: 358513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10984.168055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10984.755636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10989.485457] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11000.608531] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11010.246919] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11022.300199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11022.765348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11029.991914] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11036.107974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11039.984612] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11051.491648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11051.496924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11051.507762] Lustre: Skipped 1 previous similar message [11051.519236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11051.522032] Lustre: Skipped 2 previous similar messages [11053.621596] Lustre: server umount lustre-MDT0000 complete [11056.616048] LustreError: 359240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11056.647034] LustreError: 359240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11057.362422] LustreError: 358492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787803998 with bad export cookie 10100834999888842982 [11057.376692] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11057.376781] LustreError: 358492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11057.778283] Lustre: server umount lustre-MDT0001 complete [11072.184606] Lustre: server umount lustre-OST0000 complete [11089.350893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11089.797212] LustreError: 362226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11089.853889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11095.015695] LustreError: 362227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11096.196791] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11105.262782] LustreError: 362226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11105.290705] LustreError: 362226:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11105.781706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11106.243618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11111.189875] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11118.109858] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11123.729796] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11131.646882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11131.925762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11135.043302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11138.263643] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11146.515178] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11160.554035] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11160.559161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11160.561757] Lustre: Skipped 2 previous similar messages [11160.596121] Lustre: Skipped 2 previous similar messages [11164.723912] Lustre: server umount lustre-MDT0000 complete [11165.686439] LustreError: 364232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11165.695566] LustreError: 364232:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11168.442951] LustreError: 364907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804109 with bad export cookie 10100834999888844095 [11168.454085] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11168.460576] LustreError: 364907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11168.656146] Lustre: server umount lustre-MDT0001 complete [11184.595696] Lustre: server umount lustre-OST0000 complete [11204.156126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11204.752571] LustreError: 366138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11204.830648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11210.760213] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11221.337332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11226.632703] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11235.792737] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11243.275621] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11253.088915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11254.306409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11254.314776] Lustre: Skipped 1 previous similar message [11263.215982] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11274.586906] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11274.737307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11281.894113] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11281.900654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11281.914649] Lustre: Skipped 1 previous similar message [11281.923277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11284.965304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11284.967353] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11284.997726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11287.799714] Lustre: server umount lustre-OST0000 complete [11293.665796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11293.671173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11293.685224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11293.688256] Lustre: Skipped 1 previous similar message [11297.618650] Lustre: server umount lustre-MDT0000 complete [11300.324556] LustreError: 366138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11300.350967] LustreError: 366138:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11300.661066] LustreError: 366119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804241 with bad export cookie 10100834999888845117 [11300.663341] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11300.671720] LustreError: 366119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11300.978644] Lustre: server umount lustre-MDT0001 complete [11309.746610] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [11312.251647] Key type lgssc unregistered [11312.552574] LNet: 369677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11312.568588] LNetError: 369677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11313.640608] LNet: Removed LNI 192.168.204.136@tcp [11314.552387] Key type .llcrypt unregistered [11314.554224] Key type ._llcrypt unregistered [11341.885842] Key type ._llcrypt registered [11341.888667] Key type .llcrypt registered [11342.025563] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [11343.593964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11343.665428] alg: No test for adler32 (adler32-zlib) [11344.723877] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [11344.977808] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [11346.704234] Key type lgssc registered [11347.960532] Lustre: Echo OBD driver; http://www.lustre.org/ [11356.988088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11357.014825] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11357.022054] LustreError: 371138:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11357.041372] LustreError: 371138:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11357.046597] LustreError: 371138:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11357.050337] LustreError: 371138:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11357.056093] LustreError: 371138:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11364.933735] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11364.950681] LustreError: 371535:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11364.988918] LustreError: 371535:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11364.999281] LustreError: 371535:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11365.009275] LustreError: 371535:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11365.028976] LustreError: 371535:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11386.617981] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [11389.602969] Key type lgssc unregistered [11389.989877] LNet: 372543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11390.004393] LNetError: 372543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11390.026040] LNet: Removed LNI 192.168.204.136@tcp [11391.325525] Key type .llcrypt unregistered [11391.327989] Key type ._llcrypt unregistered [11411.251582] Key type ._llcrypt registered [11411.253993] Key type .llcrypt registered [11411.401728] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [11426.154360] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [11427.206496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11427.419924] alg: No test for adler32 (adler32-zlib) [11428.597720] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [11428.837200] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [11430.512179] Key type lgssc registered [11431.734676] Lustre: Echo OBD driver; http://www.lustre.org/ [11464.458369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11464.494512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11465.737790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11465.767665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11465.843527] Lustre: lustre-MDT0000: new disk, initializing [11465.913462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11465.932448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11470.923181] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11480.846734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11480.977556] Lustre: 375700:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11481.065539] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11481.074363] Lustre: Skipped 1 previous similar message [11481.145410] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11481.195367] Lustre: lustre-MDT0001: new disk, initializing [11481.270135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11481.335138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11481.352854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11485.804277] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11495.410315] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11499.832279] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11505.387682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11505.621992] Lustre: lustre-OST0000: new disk, initializing [11505.629936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11505.639422] Lustre: 377667:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11505.725918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11510.072843] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11512.891402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11512.900857] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11512.989689] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11517.659054] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11521.251607] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11521.401230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11525.095867] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11525.277332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11528.678683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11528.683329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11528.690469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11528.694027] Lustre: Skipped 1 previous similar message [11533.804627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11533.811614] Lustre: Skipped 2 previous similar messages [11538.915100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11540.961041] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11541.119338] Lustre: server umount lustre-OST0000 complete [11548.135901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11548.145489] LustreError: Skipped 1 previous similar message [11548.151314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11548.165609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11548.169466] Lustre: Skipped 1 previous similar message [11549.155162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11550.439975] Lustre: server umount lustre-MDT0000 complete [11553.190310] LustreError: 375692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804493 with bad export cookie 2035224136269725858 [11553.191376] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11553.197761] LustreError: 375692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11553.398074] Lustre: server umount lustre-MDT0001 complete [11560.738708] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11562.199424] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 00:21:40 (1787804500) [11569.530207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11570.281753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11575.779535] LustreError: 380126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11575.790367] LustreError: 380126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11576.009077] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11579.878514] LustreError: 380127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11583.129876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11583.353943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11586.683671] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11592.126408] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11595.856691] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11600.553054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11600.773520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11604.821364] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11609.709498] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11614.378622] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11614.536310] Lustre: lustre-OST0001: new disk, initializing [11614.541823] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11614.550306] Lustre: 382960:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11618.353817] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11619.857833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11619.863108] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11619.933082] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11625.293751] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11630.058428] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11630.058666] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11630.062115] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11630.062126] Lustre: Skipped 2 previous similar messages [11630.095273] Lustre: Skipped 1 previous similar message [11634.214214] Lustre: server umount lustre-OST0001 complete [11635.185895] LustreError: 383044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11635.193596] LustreError: 383044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11640.291534] LustreError: 383044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11640.301244] LustreError: 383044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11645.414532] LustreError: 383044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11645.424898] LustreError: 383044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11650.021120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11650.028659] LustreError: Skipped 1 previous similar message [11650.032960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11650.041801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11650.048205] Lustre: Skipped 2 previous similar messages [11655.498949] Lustre: server umount lustre-OST0000 complete [11660.258815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11660.265339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11660.280909] Lustre: Skipped 1 previous similar message [11664.799701] Lustre: server umount lustre-MDT0000 complete [11665.889266] LustreError: 380127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11665.901894] LustreError: 380127:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11667.377841] LustreError: 380107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804608 with bad export cookie 2035224136269726740 [11667.380458] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11667.385649] LustreError: 380107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11673.772392] Lustre: server umount lustre-MDT0001 complete [11682.079602] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [11683.891284] Key type lgssc unregistered [11684.166801] LNet: 384948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11684.173630] LNetError: 384948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11684.185153] LNet: Removed LNI 192.168.204.136@tcp [11684.841252] Key type .llcrypt unregistered [11684.844192] Key type ._llcrypt unregistered [11719.034627] Key type ._llcrypt registered [11719.036471] Key type .llcrypt registered [11719.134713] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [11719.809705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11719.827827] alg: No test for adler32 (adler32-zlib) [11720.786452] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [11720.963178] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [11722.568175] Key type lgssc registered [11723.357787] Lustre: Echo OBD driver; http://www.lustre.org/ [11729.725410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11729.744121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11730.943866] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11730.963303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11731.106260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11734.775776] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11741.976526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11742.046068] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11742.081387] Lustre: 387724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11742.322228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11745.846359] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11751.943734] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11756.505308] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11758.050649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11758.056114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11758.065892] Lustre: Skipped 1 previous similar message [11758.081940] Lustre: Skipped 1 previous similar message [11763.170436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11763.174219] Lustre: Skipped 1 previous similar message [11768.294239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11768.297102] Lustre: Skipped 1 previous similar message [11772.384977] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11772.487451] Lustre: server umount lustre-MDT0000 complete [11775.667722] LustreError: 387717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804716 with bad export cookie 11107549049666463209 [11775.675581] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11775.877194] Lustre: server umount lustre-MDT0001 complete [11783.392656] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 00:25:22 (1787804722) [11788.643524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11789.044383] LustreError: 390239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11789.055596] LustreError: 390239:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11789.092697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11792.412329] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11794.405474] LustreError: 390240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11799.182937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11799.496246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11803.423628] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11811.160837] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11817.422895] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11824.716721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11824.886644] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11825.061364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11831.567217] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11838.564250] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11845.755815] Lustre: Permanently deactivating lustre-OST0000 [11845.787151] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11853.324444] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11855.603427] LustreError: 393226:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11855.617722] LustreError: 393226:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11855.629366] LustreError: 393226:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11858.274985] LustreError: 393275:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11858.283324] LustreError: 393275:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11866.455599] Lustre: server umount lustre-OST0000 complete [11870.107686] LustreError: 393526:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11870.121381] LustreError: 393526:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11872.229811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11872.253421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11877.346904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11877.366737] Lustre: Skipped 2 previous similar messages [11881.495275] Lustre: server umount lustre-MDT0000 complete [11882.466787] LustreError: 390240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11882.525797] LustreError: 390240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11885.490282] LustreError: 392930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804826 with bad export cookie 11107549049666463790 [11885.492470] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11885.501693] LustreError: 392930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11895.992460] Lustre: server umount lustre-MDT0001 complete [11904.938381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11909.370251] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11912.284201] LustreError: 394685:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11919.149726] Lustre: 395024:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.136@tcp [11921.554469] Lustre: server umount MGS complete [11933.743516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11934.406921] LustreError: 395748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11934.437917] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11934.443922] Lustre: Skipped 1 previous similar message [11934.498957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11939.812870] LustreError: 395749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11939.857111] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11947.895088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11948.168908] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11952.281743] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11959.480597] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11966.399983] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11968.495363] Lustre: Permanently reactivating lustre-OST0000 [11968.504430] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11968.510216] Lustre: Skipped 2 previous similar messages [11999.204091] LustreError: 397581:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12006.812605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12007.138610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12007.142220] Lustre: Skipped 1 previous similar message [12012.744672] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12020.183359] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12027.374866] Lustre: server umount lustre-OST0000 complete [12035.044235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12035.059262] Lustre: Skipped 1 previous similar message [12035.061324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12035.071354] Lustre: Skipped 1 previous similar message [12042.306740] Lustre: server umount lustre-MDT0000 complete [12044.258817] Lustre: 387048:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787804969/real 1787804969] req@00000000c29eea07 x1874649060308352/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787804985 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12044.301610] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12044.311773] LustreError: 396477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12044.325838] LustreError: 396477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12045.283376] Lustre: 387050:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787804970/real 1787804970] req@00000000c123a6cf x1874649060309120/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787804986 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12046.024036] LustreError: 398476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804986 with bad export cookie 11107549049666464840 [12046.026381] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12046.031602] LustreError: 398476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12046.394249] Lustre: server umount lustre-MDT0001 complete [12056.569739] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [12059.255677] Key type lgssc unregistered [12059.522076] LNet: 400166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12059.529943] LNetError: 400166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12060.588460] LNet: Removed LNI 192.168.204.136@tcp [12061.362293] Key type .llcrypt unregistered [12061.364248] Key type ._llcrypt unregistered [12078.081790] Key type ._llcrypt registered [12078.084878] Key type .llcrypt registered [12078.186706] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [12092.278235] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [12093.172533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12093.202365] alg: No test for adler32 (adler32-zlib) [12094.353585] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [12094.622766] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [12096.296476] Key type lgssc registered [12097.618645] Lustre: Echo OBD driver; http://www.lustre.org/ [12135.442170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12135.472739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12136.749091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12136.806729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12136.882210] Lustre: lustre-MDT0000: new disk, initializing [12136.953404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12136.975612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12141.490921] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12153.412847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12153.567251] Lustre: 403322:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12153.657434] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12153.662538] Lustre: Skipped 1 previous similar message [12153.756235] Lustre: lustre-MDT0001: new disk, initializing [12153.869247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12153.931689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12153.944542] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12159.330893] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12169.499273] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12175.774172] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12182.511342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12182.931692] Lustre: lustre-OST0000: new disk, initializing [12182.936250] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12182.943280] Lustre: 405292:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12183.082172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12189.779508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12189.793672] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12189.877045] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12189.901962] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12199.884350] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12205.266556] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12205.494624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12210.489577] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12210.642535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12215.780283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12215.781540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12215.784774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12215.802318] Lustre: Skipped 1 previous similar message [12215.822510] Lustre: Skipped 1 previous similar message [12219.096059] Lustre: server umount lustre-OST0000 complete [12225.506846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12225.514106] LustreError: Skipped 1 previous similar message [12225.519091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12225.534929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12229.055849] Lustre: server umount lustre-MDT0000 complete [12232.390076] LustreError: 403315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787805173 with bad export cookie 6300002072108290460 [12232.399062] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12232.404205] LustreError: 403315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12232.693748] Lustre: server umount lustre-MDT0001 complete [12238.951880] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12240.183810] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 00:32:59 (1787805179) [12245.607420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12245.936260] LustreError: 407753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12245.949735] LustreError: 407753:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12245.998889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12250.609916] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12251.114682] LustreError: 407754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12256.232476] LustreError: 407753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12258.643354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12258.960306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12262.878099] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12269.290935] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12274.511691] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12280.820697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12281.157189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12287.257122] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12293.864687] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12295.694121] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12298.255455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12304.867383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12304.874144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12304.880823] Lustre: Skipped 1 previous similar message [12304.884292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12304.887818] Lustre: Skipped 2 previous similar messages [12306.918543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12306.918574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12306.921528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12306.937064] Lustre: Skipped 1 previous similar message [12308.537614] Lustre: server umount lustre-OST0000 complete [12312.035186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12312.040103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12312.043388] Lustre: Skipped 1 previous similar message [12312.057932] Lustre: Skipped 1 previous similar message [12322.277399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12322.288626] Lustre: Skipped 3 previous similar messages [12326.368330] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12326.555746] Lustre: server umount lustre-MDT0000 complete [12327.400375] LustreError: 408481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12327.417960] LustreError: 408481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12329.548363] LustreError: 408488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787805270 with bad export cookie 6300002072108291342 [12329.551584] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12329.554763] LustreError: 408488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12329.835079] Lustre: server umount lustre-MDT0001 complete [12339.176644] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [12341.015567] Key type lgssc unregistered [12341.256487] LNet: 411482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12341.268927] LNetError: 411482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12342.311384] LNet: Removed LNI 192.168.204.136@tcp [12342.948628] Key type .llcrypt unregistered [12342.951613] Key type ._llcrypt unregistered [12361.889190] Key type ._llcrypt registered [12361.891595] Key type .llcrypt registered [12361.981546] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [12362.199714] Key type .llcrypt unregistered [12362.204562] Key type ._llcrypt unregistered [12376.198764] Key type ._llcrypt registered [12376.202468] Key type .llcrypt registered [12376.274844] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [12376.924318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12376.939556] alg: No test for adler32 (adler32-zlib) [12377.869957] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [12378.017954] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [12379.648558] Key type lgssc registered [12380.295169] Lustre: Echo OBD driver; http://www.lustre.org/ [12384.852490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12384.865811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12386.132123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12388.974809] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12395.493866] LustreError: 413692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12395.511127] LustreError: 413692:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12395.526690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12395.790664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12400.391478] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12401.129168] LustreError: 413693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12401.153497] LustreError: 413693:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12406.246610] LustreError: 413692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12406.262923] LustreError: 413692:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12406.540851] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12410.340632] LustreError: 413693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12410.352616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12412.008220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12412.312624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12415.428577] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12427.756448] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12427.760215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12427.762765] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12427.773398] Lustre: Skipped 1 previous similar message [12427.789221] Lustre: Skipped 1 previous similar message [12431.873814] Lustre: server umount lustre-OST0000 complete [12434.470963] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12434.481399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12434.497730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12437.986510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12437.986663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12438.005205] Lustre: Skipped 1 previous similar message [12440.719953] Lustre: server umount lustre-MDT0000 complete [12443.106208] LustreError: 415426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12443.117471] LustreError: 415426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12443.626685] LustreError: 413673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787805384 with bad export cookie 5758068107424381562 [12443.635876] LustreError: 413673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12443.638409] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12443.862126] Lustre: server umount lustre-MDT0001 complete [12452.045533] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [12454.253319] Key type lgssc unregistered [12454.517093] LNet: 416839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12454.524360] LNetError: 416839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12454.539303] LNet: Removed LNI 192.168.204.136@tcp [12455.278950] Key type .llcrypt unregistered [12455.282826] Key type ._llcrypt unregistered [12472.625962] Key type ._llcrypt registered [12472.628556] Key type .llcrypt registered [12472.693616] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [12473.295498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12473.361663] alg: No test for adler32 (adler32-zlib) [12474.374515] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [12474.551937] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [12476.225772] Key type lgssc registered [12477.174029] Lustre: Echo OBD driver; http://www.lustre.org/ [12482.762792] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12482.809709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12498.336375] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12503.456336] LustreError: 418220:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.136@tcp: failed processing log, type 4: rc = -110 [12530.144946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12534.717309] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12540.517419] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12545.564830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12545.892618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12545.954709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12548.844832] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12551.140761] LustreError: 418245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12551.152539] LustreError: 418245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12555.046440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12555.307177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12558.703887] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12563.765857] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12567.760742] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12574.694435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12574.698425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12574.708137] Lustre: Skipped 1 previous similar message [12578.666484] Lustre: server umount lustre-OST0000 complete [12584.929514] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12584.934017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12584.942340] Lustre: Skipped 1 previous similar message [12584.953369] Lustre: Skipped 2 previous similar messages [12587.847832] Lustre: server umount lustre-MDT0000 complete [12590.049685] LustreError: 419255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12590.063097] LustreError: 419255:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12591.101176] LustreError: 418227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787805531 with bad export cookie 18381845607880153456 [12591.109322] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12591.118396] LustreError: 418227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12591.393604] Lustre: server umount lustre-MDT0001 complete [12600.702334] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [12603.278218] Key type lgssc unregistered [12603.582180] LNet: 421791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12603.597637] LNetError: 421791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12603.617316] LNet: Removed LNI 192.168.204.136@tcp [12604.467973] Key type .llcrypt unregistered [12604.471251] Key type ._llcrypt unregistered [12625.408504] Key type ._llcrypt registered [12625.410861] Key type .llcrypt registered [12625.492636] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [12626.179037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12626.264367] alg: No test for adler32 (adler32-zlib) [12627.396847] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [12627.642113] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [12629.296775] Key type lgssc registered [12630.294442] Lustre: Echo OBD driver; http://www.lustre.org/ [12636.970699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12636.993588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12638.339507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12642.305572] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12648.931422] LustreError: 423188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12648.956142] LustreError: 423188:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12650.474281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12650.878321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12654.740219] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12661.535956] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12667.526077] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12675.153814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12675.538788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12682.292883] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12683.762101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12690.128651] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12697.060465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12697.070784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12697.089157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12698.202243] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.36@tcp (stopping) [12699.109879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12699.130065] Lustre: Skipped 1 previous similar message [12701.786308] Lustre: server umount lustre-MDT0000 complete [12703.311398] LustreError: 423184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12704.358870] LustreError: 423169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.36@tcp arrived at 1787805645 with bad export cookie 8885933511694673276 [12704.391288] LustreError: 423169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12706.308339] LustreError: 423167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787805647 with bad export cookie 8885933511694672380 [12706.322377] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12706.330028] LustreError: 423167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12707.299688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12707.312376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12707.335591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12707.351393] Lustre: Skipped 3 previous similar messages [12709.346456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12709.348305] LustreError: 423187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12709.375580] LustreError: 423187:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12711.393788] LustreError: 423915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12711.419540] LustreError: 423915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12712.711794] Lustre: server umount lustre-OST0000 complete [12716.513805] LustreError: 423187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12716.530312] LustreError: 423187:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12719.399297] Lustre: server umount lustre-MDT0001 complete [12730.257659] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [12732.771212] Key type lgssc unregistered [12733.112248] LNet: 426875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12733.128475] LNetError: 426875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12733.172959] LNet: Removed LNI 192.168.204.136@tcp [12734.207953] Key type .llcrypt unregistered [12734.211306] Key type ._llcrypt unregistered [12758.610371] Key type ._llcrypt registered [12758.612686] Key type .llcrypt registered [12758.742485] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [12759.667156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12759.862738] alg: No test for adler32 (adler32-zlib) [12761.019274] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [12761.254358] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [12762.914511] Key type lgssc registered [12764.475272] Lustre: Echo OBD driver; http://www.lustre.org/ [12774.669623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12774.697759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12776.078602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12780.830941] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12785.637098] LustreError: 428292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12785.684266] LustreError: 428292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12790.029816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12790.474945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12795.036681] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12801.958798] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12808.567074] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12816.598334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12816.881254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12820.993142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12825.659453] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12833.476256] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12840.418190] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12840.438096] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12840.447331] Lustre: Skipped 1 previous similar message [12841.569388] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.36@tcp (stopping) [12844.370733] Lustre: server umount lustre-MDT0001 complete [12845.559256] LustreError: 428293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12845.589184] LustreError: 428293:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12846.658463] LustreError: 428288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12850.666148] LustreError: 428293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12850.699278] LustreError: 428293:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12855.788282] LustreError: 428292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12855.797526] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12855.820143] Lustre: Skipped 1 previous similar message [12855.830604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12859.307666] Lustre: server umount lustre-OST0000 complete [12863.570853] Lustre: server umount lustre-MDT0000 complete [12878.526460] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [12881.953882] Key type lgssc unregistered [12882.224853] LNet: 431980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12882.244567] LNetError: 431980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12882.267526] LNet: Removed LNI 192.168.204.136@tcp [12883.274979] Key type .llcrypt unregistered [12883.278317] Key type ._llcrypt unregistered [12905.646343] Key type ._llcrypt registered [12905.648492] Key type .llcrypt registered [12905.736357] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [12919.839760] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [12920.993370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12921.031621] alg: No test for adler32 (adler32-zlib) [12922.097762] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [12922.363092] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [12924.041748] Key type lgssc registered [12925.127726] Lustre: Echo OBD driver; http://www.lustre.org/ [12958.602759] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [12973.082050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12973.115888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12974.350572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12974.421662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12974.497709] Lustre: lustre-MDT0000: new disk, initializing [12974.590409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12974.610351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12978.912582] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12993.900847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12994.147844] Lustre: lustre-OST0000: new disk, initializing [12994.152165] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12994.158613] Lustre: Skipped 1 previous similar message [12994.163481] Lustre: 437014:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12994.242337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13000.145784] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [13001.330492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13001.337496] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13001.367979] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13008.490188] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13024.611623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13024.754299] Lustre: 435994:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13024.825223] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13024.891933] Lustre: lustre-MDT0001: new disk, initializing [13024.988846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13025.029976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13025.040219] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13030.398760] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [13041.242153] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13041.463703] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13045.793795] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13046.041432] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13050.968217] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.36@tcp (stopping) [13051.363872] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13051.372847] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13056.055899] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.36@tcp (stopping) [13056.066261] Lustre: Skipped 2 previous similar messages [13056.346795] Lustre: server umount lustre-MDT0001 complete [13061.604562] LustreError: 436008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13061.614910] LustreError: 436008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13066.725898] LustreError: 436003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13066.746610] LustreError: 436003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13071.844822] LustreError: 436003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13071.862639] LustreError: 436003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13076.964859] LustreError: 436007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13076.966977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13077.008236] Lustre: Skipped 1 previous similar message [13077.023893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13080.032493] Lustre: server umount lustre-OST0000 complete [13082.086915] LustreError: 436008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13091.552185] Lustre: server umount lustre-MDT0000 complete [13106.299744] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [13108.861051] Key type lgssc unregistered [13109.140249] LNet: 440767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13109.148956] LNetError: 440767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13109.174969] LNet: Removed LNI 192.168.204.136@tcp [13110.216363] Key type .llcrypt unregistered [13110.218128] Key type ._llcrypt unregistered [13151.592545] Key type ._llcrypt registered [13151.594793] Key type .llcrypt registered [13151.691422] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [13152.038545] Key type .llcrypt unregistered [13152.042356] Key type ._llcrypt unregistered [13169.320718] Key type ._llcrypt registered [13169.321956] Key type .llcrypt registered [13169.403770] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [13170.309797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13170.322155] alg: No test for adler32 (adler32-zlib) [13171.351108] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [13171.602619] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [13173.241318] Key type lgssc registered [13174.264447] Lustre: Echo OBD driver; http://www.lustre.org/ [13208.916806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13208.976042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13210.343499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13210.373970] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13210.487383] Lustre: lustre-MDT0000: new disk, initializing [13210.584092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13210.603470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13216.127489] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [13227.937255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13228.069160] Lustre: 444149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13228.153953] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13228.160099] Lustre: Skipped 1 previous similar message [13228.267283] Lustre: lustre-MDT0001: new disk, initializing [13228.452567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13228.502420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13228.538233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13233.561736] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [13242.492429] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13248.177403] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13254.528947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13254.748399] Lustre: lustre-OST0000: new disk, initializing [13254.751839] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13254.756880] Lustre: 446116:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13254.835708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13260.343367] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [13264.470354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13264.484576] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13264.599416] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13269.076614] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13279.713426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13279.717881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13279.725358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13280.228497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13280.229819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13280.250146] Lustre: Skipped 1 previous similar message [13280.256367] Lustre: Skipped 1 previous similar message [13281.447340] Lustre: server umount lustre-MDT0000 complete [13284.647248] LustreError: 444141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787806225 with bad export cookie 548342597971063929 [13284.659628] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13284.677036] LustreError: 444141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13284.939501] Lustre: server umount lustre-MDT0001 complete [13299.062446] Lustre: server umount lustre-OST0000 complete [13315.993497] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 00:50:53 (1787806253) [13324.340246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13324.896056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13329.489963] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [13329.890287] LustreError: 448145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13329.919165] LustreError: 448145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13335.011162] LustreError: 448146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13338.672933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13338.962294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13343.091071] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [13350.802917] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13355.968580] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13375.290037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13375.635973] Lustre: Found index 0 for lustre-OST0000, updating log [13375.904225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13382.849535] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [13390.951181] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13395.347574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13404.643630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13404.655896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13404.678870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13404.697996] Lustre: Skipped 1 previous similar message [13407.520554] Lustre: server umount lustre-OST0000 complete [13426.145205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13426.149064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13426.156824] Lustre: Skipped 1 previous similar message [13426.163683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13426.168680] Lustre: Skipped 3 previous similar messages [13429.530806] Lustre: server umount lustre-MDT0000 complete [13432.803399] LustreError: 448146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13432.840621] LustreError: 448146:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13433.583710] LustreError: 448126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787806374 with bad export cookie 548342597971065322 [13433.601026] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13433.612368] LustreError: 448126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13434.039978] Lustre: server umount lustre-MDT0001 complete [13435.808548] LNet: 1 local NIs in recovery (showing 1): 192.168.204.136@tcp [13444.129318] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 00:53:02 (1787806382) [14071.760574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14072.295561] LustreError: 454572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14072.396617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14077.427042] LustreError: 454573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14079.233554] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [14082.531666] LustreError: 454572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14088.678282] LustreError: 454573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14088.693444] LustreError: 454573:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14089.284085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14089.748541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14094.855864] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [14104.529762] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14112.458564] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14127.009754] LNet: 1 local NIs in recovery (showing 1): 192.168.204.136@tcp [14132.540844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14132.811242] Lustre: Found index 0 for lustre-OST0000, updating log [14133.213051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14138.419433] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [14147.921513] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14151.862399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14151.876196] Lustre: Skipped 1 previous similar message [14151.881505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14151.895841] Lustre: Skipped 2 previous similar messages [14157.390883] Lustre: server umount lustre-OST0000 complete [14163.177757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14163.212535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14163.227087] Lustre: Skipped 1 previous similar message [14164.258894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14166.307219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14166.315204] Lustre: Skipped 4 previous similar messages [14168.496888] Lustre: server umount lustre-MDT0000 complete [14169.403265] LustreError: 454572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14169.426023] LustreError: 454572:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14172.841059] LustreError: 454554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787807113 with bad export cookie 548342597971066400 [14172.843844] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14172.849096] LustreError: 454554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14173.195394] Lustre: server umount lustre-MDT0001 complete [14186.400158] LNet: 1 local NIs in recovery (showing 1): 192.168.204.136@tcp [14203.363747] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [14212.633667] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [14249.879301] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 01:06:28 (1787807188) [14268.542238] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [14271.321431] Key type lgssc unregistered [14271.699484] LNet: 461209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14271.706152] LNetError: 461209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14394.600922] LNet: Removed LNI 192.168.204.136@tcp [14395.442275] Key type .llcrypt unregistered [14395.448755] Key type ._llcrypt unregistered [14413.382951] Key type ._llcrypt registered [14413.390281] Key type .llcrypt registered [14413.539859] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [14414.950095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14414.988922] alg: No test for adler32 (adler32-zlib) [14416.073306] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [14416.367896] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [14418.176292] Key type lgssc registered [14419.558572] Lustre: Echo OBD driver; http://www.lustre.org/ [14424.509529] LNet: Added LNI 192.168.204.136@tcp100 [8/256/0/180] [14425.416447] LNet: Added LNI 192.168.204.136@tcp101 [8/256/0/180] [14426.207668] LNet: Added LNI 192.168.204.136@tcp102 [8/256/0/180] [14427.094044] LNet: Added LNI 192.168.204.136@tcp103 [8/256/0/180] [14428.226351] LNet: Added LNI 192.168.204.136@tcp104 [8/256/0/180] [14429.214800] LNet: Added LNI 192.168.204.136@tcp105 [8/256/0/180] [14430.107584] LNet: Added LNI 192.168.204.136@tcp106 [8/256/0/180] [14431.205647] LNet: Added LNI 192.168.204.136@tcp107 [8/256/0/180] [14431.998307] LNet: Added LNI 192.168.204.136@tcp108 [8/256/0/180] [14432.949639] LNet: Added LNI 192.168.204.136@tcp109 [8/256/0/180] [14433.889504] LNet: Added LNI 192.168.204.136@tcp110 [8/256/0/180] [14434.845642] LNet: Added LNI 192.168.204.136@tcp111 [8/256/0/180] [14435.896266] LNet: Added LNI 192.168.204.136@tcp112 [8/256/0/180] [14436.910760] LNet: Added LNI 192.168.204.136@tcp113 [8/256/0/180] [14437.973490] LNet: Added LNI 192.168.204.136@tcp114 [8/256/0/180] [14438.943687] LNet: Added LNI 192.168.204.136@tcp115 [8/256/0/180] [14439.794550] LNet: Added LNI 192.168.204.136@tcp116 [8/256/0/180] [14440.609449] LNet: Added LNI 192.168.204.136@tcp117 [8/256/0/180] [14441.656509] LNet: Added LNI 192.168.204.136@tcp118 [8/256/0/180] [14442.711155] LNet: Added LNI 192.168.204.136@tcp119 [8/256/0/180] [14443.567757] LNet: Added LNI 192.168.204.136@tcp120 [8/256/0/180] [14444.626733] LNet: Added LNI 192.168.204.136@tcp121 [8/256/0/180] [14445.790325] LNet: Added LNI 192.168.204.136@tcp122 [8/256/0/180] [14446.790430] LNet: Added LNI 192.168.204.136@tcp123 [8/256/0/180] [14447.685495] LNet: Added LNI 192.168.204.136@tcp124 [8/256/0/180] [14448.684454] LNet: Added LNI 192.168.204.136@tcp125 [8/256/0/180] [14449.673809] LNet: Added LNI 192.168.204.136@tcp126 [8/256/0/180] [14450.652777] LNet: Added LNI 192.168.204.136@tcp127 [8/256/0/180] [14451.537161] LNet: Added LNI 192.168.204.136@tcp128 [8/256/0/180] [14452.414555] LNet: Added LNI 192.168.204.136@tcp129 [8/256/0/180] [14453.498372] LNet: Added LNI 192.168.204.136@tcp130 [8/256/0/180] [14454.239923] LNet: Added LNI 192.168.204.136@tcp131 [8/256/0/180] [14455.234296] LNet: Added LNI 192.168.204.136@tcp132 [8/256/0/180] [14456.245969] LNet: Added LNI 192.168.204.136@tcp133 [8/256/0/180] [14457.026316] LNet: Added LNI 192.168.204.136@tcp134 [8/256/0/180] [14457.827159] LNet: Added LNI 192.168.204.136@tcp135 [8/256/0/180] [14465.816458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14465.852455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14467.108376] Lustre: 464245:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.136@tcp: can't add NID 192.168.204.136@tcp130: rc = -75 [14467.191480] LNetError: 464265:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.136 [14467.215942] LNetError: Protocol error connecting to 192.168.204.136@tcp122 at host 192.168.204.136:988: is it running a compatible version of Lustre? [14467.440381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14467.513757] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14467.616124] LNet: 7 local NIs in recovery (showing 7): 192.168.204.136@tcp122, 192.168.204.136@tcp124, 192.168.204.136@tcp127, 192.168.204.136@tcp126, 192.168.204.136@tcp125, 192.168.204.136@tcp130, 192.168.204.136@tcp129 [14467.722345] Lustre: lustre-MDT0000: new disk, initializing [14467.822716] LNetError: 464270:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.136 [14467.846579] Lustre: 464301:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.136@tcp: can't add NID 192.168.204.136@tcp130: rc = -75 [14467.853729] LNetError: 464270:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [14467.853879] LNetError: Protocol error connecting to 192.168.204.136@tcp131 at host 192.168.204.136:988: is it running a compatible version of Lustre? [14467.853883] LNetError: Skipped 10 previous similar messages [14467.892983] Lustre: 464301:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14467.915716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14467.934530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14470.757274] LNetError: 461703:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.136 [14470.762752] LNetError: 461703:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 14 previous similar messages [14470.769545] LNetError: Protocol error connecting to 192.168.204.136@tcp122 at host 192.168.204.136:988: is it running a compatible version of Lustre? [14470.774520] LNetError: Skipped 16 previous similar messages [14473.266885] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [14474.850668] LNetError: 464270:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.136 [14474.862971] LNetError: 464270:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14474.871688] LNetError: Protocol error connecting to 192.168.204.136@tcp122 at host 192.168.204.136:988: is it running a compatible version of Lustre? [14474.896224] LNetError: Skipped 3 previous similar messages [14482.034316] LNetError: 464266:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.136 [14482.060235] LNetError: Protocol error connecting to 192.168.204.136@tcp122 at host 192.168.204.136:988: is it running a compatible version of Lustre? [14484.522333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14484.647392] Lustre: 464261:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14484.699156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14484.704276] Lustre: Skipped 1 previous similar message [14484.781740] Lustre: lustre-MDT0001: new disk, initializing [14484.826844] Lustre: 465063:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.136@tcp: can't add NID 192.168.204.136@tcp130: rc = -75 [14484.839713] Lustre: 465063:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14484.981518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14485.021445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14485.040814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14490.038590] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [14491.240355] LNetError: 464266:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.136 [14491.259985] LNetError: 464266:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [14491.268459] LNetError: Protocol error connecting to 192.168.204.136@tcp133 at host 192.168.204.136:988: is it running a compatible version of Lustre? [14491.281977] LNetError: Skipped 9 previous similar messages [14500.031518] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14506.815729] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14509.544451] LNetError: 464271:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.136 [14509.562892] LNetError: 464271:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 14 previous similar messages [14509.573824] LNetError: Protocol error connecting to 192.168.204.136@tcp130 at host 192.168.204.136:988: is it running a compatible version of Lustre? [14509.599076] LNetError: Skipped 14 previous similar messages [14514.140538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14514.451558] Lustre: lustre-OST0000: new disk, initializing [14514.462169] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14514.473130] Lustre: 466237:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14514.584519] Lustre: 466242:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.136@tcp: can't add NID 192.168.204.136@tcp130: rc = -75 [14514.593845] Lustre: 466242:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [14514.654350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14521.320512] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [14522.936064] Lustre: 466666:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.136@tcp: can't add NID 192.168.204.136@tcp130: rc = -75 [14522.948329] Lustre: 466666:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14522.997729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14523.005647] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14523.115265] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14528.096106] LNet: 2 local NIs in recovery (showing 2): 192.168.204.136@tcp133, 192.168.204.136@tcp122 [14531.928174] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14543.330910] LNetError: 461703:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.136 [14543.357190] LNetError: 461703:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 32 previous similar messages [14543.375753] LNetError: Protocol error connecting to 192.168.204.136@tcp119 at host 192.168.204.136:988: is it running a compatible version of Lustre? [14543.384686] LNetError: Skipped 32 previous similar messages [14543.850565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14543.853476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14543.857160] LustreError: Skipped 1 previous similar message [14543.862362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14543.886330] Lustre: Skipped 1 previous similar message [14545.928500] Lustre: server umount lustre-OST0000 complete [14551.524187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14551.539244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14551.551560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14551.556333] Lustre: Skipped 1 previous similar message [14554.081453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14554.082283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14554.112078] Lustre: Skipped 1 previous similar message [14556.401558] Lustre: server umount lustre-MDT0000 complete [14559.891920] LustreError: 464254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787807500 with bad export cookie 1125615731656080865 [14559.892654] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14559.901116] LustreError: 464254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14560.166847] Lustre: server umount lustre-MDT0001 complete [14570.160753] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [14573.066064] Key type lgssc unregistered [14573.491263] LNet: 468151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14573.498599] LNetError: 468151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14573.504195] LNet: Removed LNI 192.168.204.136@tcp [14574.562207] LNet: Removed LNI 192.168.204.136@tcp100 [14574.569454] LNetError: 468151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14574.584725] LNetError: 468151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14574.597240] LNet: Removed LNI 192.168.204.136@tcp101 [14575.648158] LNet: Removed LNI 192.168.204.136@tcp102 [14575.652945] LNetError: 468151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14575.660983] LNetError: 468151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14575.667596] LNet: Removed LNI 192.168.204.136@tcp103 [14576.736327] LNet: Removed LNI 192.168.204.136@tcp104 [14577.760181] LNet: Removed LNI 192.168.204.136@tcp105 [14577.766839] LNetError: 468151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14577.785537] LNetError: 468151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14577.799633] LNet: Removed LNI 192.168.204.136@tcp106 [14578.849330] LNet: Removed LNI 192.168.204.136@tcp107 [14579.872361] LNet: Removed LNI 192.168.204.136@tcp108 [14580.896437] LNet: Removed LNI 192.168.204.136@tcp109 [14581.920198] LNet: Removed LNI 192.168.204.136@tcp110 [14581.921826] LNetError: 468151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14581.930893] LNetError: 468151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14581.944393] LNet: Removed LNI 192.168.204.136@tcp111 [14583.008337] LNet: Removed LNI 192.168.204.136@tcp112 [14584.032315] LNet: Removed LNI 192.168.204.136@tcp113 [14585.123617] LNet: Removed LNI 192.168.204.136@tcp114 [14586.144164] LNet: Removed LNI 192.168.204.136@tcp115 [14587.168147] LNet: Removed LNI 192.168.204.136@tcp116 [14588.192150] LNet: Removed LNI 192.168.204.136@tcp117 [14589.216100] LNet: Removed LNI 192.168.204.136@tcp118 [14590.240172] LNet: Removed LNI 192.168.204.136@tcp119 [14590.243385] LNetError: 468151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14590.264535] LNetError: 468151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14590.281156] LNet: Removed LNI 192.168.204.136@tcp120 [14591.328280] LNet: Removed LNI 192.168.204.136@tcp121 [14591.334417] LNet: Removed LNI 192.168.204.136@tcp122 [14592.352746] LNet: Removed LNI 192.168.204.136@tcp123 [14593.377143] LNet: Removed LNI 192.168.204.136@tcp124 [14594.464329] LNet: Removed LNI 192.168.204.136@tcp125 [14595.488320] LNet: Removed LNI 192.168.204.136@tcp126 [14596.512170] LNet: Removed LNI 192.168.204.136@tcp127 [14597.540066] LNet: Removed LNI 192.168.204.136@tcp128 [14598.562640] LNet: Removed LNI 192.168.204.136@tcp129 [14599.584134] LNet: Removed LNI 192.168.204.136@tcp130 [14600.672243] LNet: Removed LNI 192.168.204.136@tcp131 [14601.696345] LNet: Removed LNI 192.168.204.136@tcp132 [14601.698363] LNet: Removed LNI 192.168.204.136@tcp133 [14602.720414] LNet: Removed LNI 192.168.204.136@tcp134 [14603.751660] LNet: Removed LNI 192.168.204.136@tcp135 [14604.479111] Key type .llcrypt unregistered [14604.481376] Key type ._llcrypt unregistered [14629.456930] Key type ._llcrypt registered [14629.464518] Key type .llcrypt registered [14629.635098] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [14648.252382] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [14649.817238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14650.015797] alg: No test for adler32 (adler32-zlib) [14651.330052] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [14651.551818] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [14653.320232] Key type lgssc registered [14654.963742] Lustre: Echo OBD driver; http://www.lustre.org/ [14695.306967] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 01:13:53 (1787807633) [14715.816656] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [14719.087175] Key type lgssc unregistered [14719.408632] LNet: 472243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14719.418575] LNetError: 472243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14719.436808] LNet: Removed LNI 192.168.204.136@tcp [14720.483536] Key type .llcrypt unregistered [14720.487075] Key type ._llcrypt unregistered [14738.685887] Key type ._llcrypt registered [14738.693394] Key type .llcrypt registered [14738.788790] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [14739.519839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14739.548262] alg: No test for adler32 (adler32-zlib) [14740.643854] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [14740.896108] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [14742.552287] Key type lgssc registered [14743.787080] Lustre: Echo OBD driver; http://www.lustre.org/ [14752.096547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14752.110793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14753.282433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14753.306611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14753.377389] Lustre: lustre-MDT0000: new disk, initializing [14753.479277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14753.518226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14758.764514] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [14770.906482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14771.116607] Lustre: 473456:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14771.193052] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14771.203772] Lustre: Skipped 1 previous similar message [14771.308486] Lustre: lustre-MDT0001: new disk, initializing [14771.428770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14771.495101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14771.515685] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14776.507660] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [14786.333747] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14792.183191] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14799.342657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14799.693239] Lustre: lustre-OST0000: new disk, initializing [14799.696517] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14799.717479] Lustre: 475426:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14799.832811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14806.101553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14806.119599] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14806.237670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14806.698372] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [14815.742418] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14820.834858] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing lnet_if_list [14827.059287] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14827.102899] LNet: Added LNI 192.168.204.136@tcp10 [8/256/0/180] [14834.559683] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14834.592710] LNet: Added LNI 192.168.204.136@tcp11 [8/256/0/180] [14839.606974] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14839.642403] LNet: Added LNI 192.168.204.136@tcp12 [8/256/0/180] [14843.979626] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14844.013717] LNet: Added LNI 192.168.204.136@tcp13 [8/256/0/180] [14848.920426] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14848.969716] LNet: Added LNI 192.168.204.136@tcp14 [8/256/0/180] [14853.904242] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14853.936717] LNet: Added LNI 192.168.204.136@tcp15 [8/256/0/180] [14858.674088] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14858.725882] LNet: Added LNI 192.168.204.136@tcp16 [8/256/0/180] [14864.406857] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14864.459758] LNet: Added LNI 192.168.204.136@tcp17 [8/256/0/180] [14870.497581] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14870.521428] LNet: Added LNI 192.168.204.136@tcp18 [8/256/0/180] [14874.886484] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14874.913989] LNet: Added LNI 192.168.204.136@tcp19 [8/256/0/180] [14879.711647] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14879.752164] LNet: Added LNI 192.168.204.136@tcp20 [8/256/0/180] [14886.044452] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14886.068722] LNet: Added LNI 192.168.204.136@tcp21 [8/256/0/180] [14892.652451] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14892.683123] LNet: Added LNI 192.168.204.136@tcp22 [8/256/0/180] [14899.068651] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14899.110614] LNet: Added LNI 192.168.204.136@tcp23 [8/256/0/180] [14905.641498] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14905.672459] LNet: Added LNI 192.168.204.136@tcp24 [8/256/0/180] [14910.921897] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14910.951939] LNet: Added LNI 192.168.204.136@tcp25 [8/256/0/180] [14916.471050] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14916.540496] LNet: Added LNI 192.168.204.136@tcp26 [8/256/0/180] [14923.291244] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14923.369091] LNet: Added LNI 192.168.204.136@tcp27 [8/256/0/180] [14930.510771] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14930.552042] LNet: Added LNI 192.168.204.136@tcp28 [8/256/0/180] [14936.422712] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14936.453344] LNet: Added LNI 192.168.204.136@tcp29 [8/256/0/180] [14942.389185] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14942.418474] LNet: Added LNI 192.168.204.136@tcp30 [8/256/0/180] [14948.937249] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14948.959945] LNet: Added LNI 192.168.204.136@tcp31 [8/256/0/180] [14955.588419] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14955.651281] LNet: Added LNI 192.168.204.136@tcp32 [8/256/0/180] [14962.865160] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14962.896374] LNet: Added LNI 192.168.204.136@tcp33 [8/256/0/180] [14969.885609] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14969.945160] LNet: Added LNI 192.168.204.136@tcp34 [8/256/0/180] [14977.172718] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14977.202143] LNet: Added LNI 192.168.204.136@tcp35 [8/256/0/180] [14984.484982] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14984.527360] LNet: Added LNI 192.168.204.136@tcp36 [8/256/0/180] [14990.996738] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14991.022928] LNet: Added LNI 192.168.204.136@tcp37 [8/256/0/180] [14998.735823] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14998.771039] LNet: Added LNI 192.168.204.136@tcp38 [8/256/0/180] [15005.915266] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15005.948399] LNet: Added LNI 192.168.204.136@tcp39 [8/256/0/180] [15013.589739] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15013.625749] LNet: Added LNI 192.168.204.136@tcp40 [8/256/0/180] [15021.476580] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15021.519341] LNet: Added LNI 192.168.204.136@tcp41 [8/256/0/180] [15029.359517] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15029.388235] LNet: Added LNI 192.168.204.136@tcp42 [8/256/0/180] [15037.953785] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15037.981836] LNet: Added LNI 192.168.204.136@tcp43 [8/256/0/180] [15045.613902] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15045.640586] LNet: Added LNI 192.168.204.136@tcp44 [8/256/0/180] [15054.809811] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15054.853400] LNet: Added LNI 192.168.204.136@tcp45 [8/256/0/180] [15064.324204] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15064.357327] LNet: Added LNI 192.168.204.136@tcp46 [8/256/0/180] [15073.917171] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15073.940629] LNet: Added LNI 192.168.204.136@tcp47 [8/256/0/180] [15083.080889] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15083.120880] LNet: Added LNI 192.168.204.136@tcp48 [8/256/0/180] [15092.589927] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15092.635374] LNet: Added LNI 192.168.204.136@tcp49 [8/256/0/180] [15103.343688] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15103.388797] LNet: Added LNI 192.168.204.136@tcp50 [8/256/0/180] [15123.941897] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15123.944101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15123.961034] Lustre: Skipped 2 previous similar messages [15123.977038] Lustre: Skipped 2 previous similar messages [15128.831730] Lustre: server umount lustre-MDT0000 complete [15132.637437] LustreError: 473448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787808073 with bad export cookie 8950930884810440060 [15132.646270] LustreError: 473448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15132.646714] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15133.163596] Lustre: server umount lustre-MDT0001 complete [15149.536221] Lustre: 472730:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787808074/real 1787808074] req@000000005ba96356 x1874652227090816/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787808090 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15149.564843] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15150.194257] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [15152.608166] Lustre: 472727:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787808077/real 1787808077] req@000000001c792db9 x1874652227090944/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787808093 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15154.721717] Lustre: 472727:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787808079/real 1787808079] req@00000000abc66ed1 x1874652227091072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787808095 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15157.379934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15158.063336] LustreError: 475418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15158.085245] LustreError: 475418:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15158.145455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15159.201249] Lustre: 472729:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787808083/real 1787808083] req@00000000d22e4cf3 x1874652227091328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787808099 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15160.161370] LustreError: 475419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15160.189210] LustreError: 475419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15162.351744] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15163.745794] Lustre: 472730:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787808088/real 1787808088] req@000000005c5763ea x1874652227091456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787808104 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15165.413638] LustreError: 499667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15165.436502] LustreError: 499667:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15170.531163] LustreError: 499665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15170.560517] LustreError: 499665:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15171.988100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15172.432382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15176.469616] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15177.710484] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15178.745734] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15183.191294] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15188.146543] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15208.419406] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15208.423820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15208.435281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15209.442404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15209.446086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15209.459750] Lustre: Skipped 1 previous similar message [15211.365814] Lustre: server umount lustre-OST0000 complete [15218.658049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15218.661973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15218.669268] Lustre: Skipped 1 previous similar message [15218.674218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15220.980064] Lustre: server umount lustre-MDT0000 complete [15224.227273] LustreError: 473450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787808164 with bad export cookie 8950930884810442748 [15224.233381] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15224.236992] LustreError: 473450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15224.499410] Lustre: server umount lustre-MDT0001 complete [15235.448266] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [15238.886044] Key type lgssc unregistered [15239.309337] LNet: 502450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15239.324748] LNetError: 502450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15239.335377] LNet: Removed LNI 192.168.204.136@tcp [15239.338727] LNet: Removed LNI 192.168.204.136@tcp10 [15239.342351] LNet: Removed LNI 192.168.204.136@tcp11 [15239.349269] LNet: Removed LNI 192.168.204.136@tcp12 [15239.357133] LNet: Removed LNI 192.168.204.136@tcp13 [15239.361790] LNet: Removed LNI 192.168.204.136@tcp14 [15239.366442] LNet: Removed LNI 192.168.204.136@tcp15 [15239.373907] LNet: Removed LNI 192.168.204.136@tcp16 [15239.381146] LNet: Removed LNI 192.168.204.136@tcp17 [15239.382816] LNet: Removed LNI 192.168.204.136@tcp18 [15239.387120] LNet: Removed LNI 192.168.204.136@tcp19 [15239.397990] LNet: Removed LNI 192.168.204.136@tcp20 [15239.401463] LNet: Removed LNI 192.168.204.136@tcp21 [15239.411394] LNet: Removed LNI 192.168.204.136@tcp22 [15239.417907] LNet: Removed LNI 192.168.204.136@tcp23 [15239.419796] LNet: Removed LNI 192.168.204.136@tcp24 [15239.425329] LNet: Removed LNI 192.168.204.136@tcp25 [15239.427826] LNet: Removed LNI 192.168.204.136@tcp26 [15239.432211] LNet: Removed LNI 192.168.204.136@tcp27 [15239.434180] LNet: Removed LNI 192.168.204.136@tcp28 [15239.438152] LNet: Removed LNI 192.168.204.136@tcp29 [15239.441219] LNet: Removed LNI 192.168.204.136@tcp30 [15239.443692] LNet: Removed LNI 192.168.204.136@tcp31 [15239.446871] LNet: Removed LNI 192.168.204.136@tcp32 [15239.451536] LNet: Removed LNI 192.168.204.136@tcp33 [15239.459560] LNet: Removed LNI 192.168.204.136@tcp34 [15239.462303] LNet: Removed LNI 192.168.204.136@tcp35 [15239.465197] LNet: Removed LNI 192.168.204.136@tcp36 [15239.467872] LNet: Removed LNI 192.168.204.136@tcp37 [15239.471225] LNet: Removed LNI 192.168.204.136@tcp38 [15239.475762] LNet: Removed LNI 192.168.204.136@tcp39 [15239.483231] LNet: Removed LNI 192.168.204.136@tcp40 [15239.490870] LNet: Removed LNI 192.168.204.136@tcp41 [15239.498390] LNet: Removed LNI 192.168.204.136@tcp42 [15239.500856] LNet: Removed LNI 192.168.204.136@tcp43 [15239.506221] LNet: Removed LNI 192.168.204.136@tcp44 [15239.511335] LNet: Removed LNI 192.168.204.136@tcp45 [15239.521299] LNet: Removed LNI 192.168.204.136@tcp46 [15239.526821] LNet: Removed LNI 192.168.204.136@tcp47 [15239.530553] LNet: Removed LNI 192.168.204.136@tcp48 [15239.534091] LNet: Removed LNI 192.168.204.136@tcp49 [15239.551318] LNet: Removed LNI 192.168.204.136@tcp50 [15240.533653] Key type .llcrypt unregistered [15240.535221] Key type ._llcrypt unregistered [15260.433118] Key type ._llcrypt registered [15260.437294] Key type .llcrypt registered [15260.550328] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [15276.728230] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [15278.045096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15278.110817] alg: No test for adler32 (adler32-zlib) [15279.139472] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [15279.527266] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [15281.208572] Key type lgssc registered [15282.416970] Lustre: Echo OBD driver; http://www.lustre.org/ [15317.836765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15317.854697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15319.078471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15319.104154] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15319.182186] Lustre: lustre-MDT0000: new disk, initializing [15319.247081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15319.262411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15323.927346] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15334.845111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15334.983479] Lustre: 505647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15335.036959] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15335.040140] Lustre: Skipped 1 previous similar message [15335.125509] Lustre: lustre-MDT0001: new disk, initializing [15335.239028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15335.299156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15335.309401] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15339.736957] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15350.335522] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15357.007256] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15365.296234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15365.570928] Lustre: *** cfs_fail_loc=90f, val=0*** [15365.702430] Lustre: lustre-OST0000: new disk, initializing [15365.706548] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15365.713067] Lustre: 507661:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15365.807222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15372.793238] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15372.894443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15372.913901] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15373.027420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15383.146705] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15393.766553] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15393.768280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15393.770839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15393.789044] LustreError: Skipped 1 previous similar message [15393.841833] Lustre: Skipped 1 previous similar message [15397.896724] Lustre: server umount lustre-OST0000 complete [15401.953621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15401.961934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15401.987470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15402.004147] Lustre: Skipped 1 previous similar message [15404.007161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15404.010321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15404.045407] Lustre: Skipped 1 previous similar message [15407.952177] Lustre: server umount lustre-MDT0000 complete [15411.949692] LustreError: 505639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787808352 with bad export cookie 15889457965786620665 [15411.951152] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15411.958074] LustreError: 505639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15412.408474] Lustre: server umount lustre-MDT0001 complete [15422.286705] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [15425.353706] Key type lgssc unregistered [15425.672171] LNet: 509574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15425.683535] LNetError: 509574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15425.707165] LNet: Removed LNI 192.168.204.136@tcp [15426.575627] Key type .llcrypt unregistered [15426.579993] Key type ._llcrypt unregistered [15447.754658] Key type ._llcrypt registered [15447.757229] Key type .llcrypt registered [15447.849916] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [15448.652727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15448.677607] alg: No test for adler32 (adler32-zlib) [15449.688164] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [15449.877862] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [15451.520390] Key type lgssc registered [15452.360906] Lustre: Echo OBD driver; http://www.lustre.org/ [15458.850519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15458.872114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15460.256788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15464.404172] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15469.554669] LustreError: 510846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15469.569772] LustreError: 510846:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15471.763197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15472.066689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15475.875231] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15483.511560] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15489.689737] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15497.908863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15498.313757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15505.069752] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15512.694729] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15525.531455] loop0: detected capacity change from 0 to 204800000 [15525.728708] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15525.737546] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15525.748711] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15531.972172] loop0: detected capacity change from 0 to 204800000 [15532.315832] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15532.337962] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15532.359562] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15537.663455] loop0: detected capacity change from 0 to 204800000 [15543.909976] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15544.014927] Lustre: fs15246-OST0000: new disk, initializing [15544.019855] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15544.033602] Lustre: 514438:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15544.100579] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15549.450789] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15558.358846] loop1: detected capacity change from 0 to 204800000 [15563.047624] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15563.147552] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15563.199917] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15563.251909] Lustre: fs15246-MDT0000: new disk, initializing [15563.418582] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15563.444273] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15563.580708] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15564.640856] Lustre: 515562:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000002101a778 x1874652970401664/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787808521 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15567.026794] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15573.741225] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15573.748735] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15573.772111] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15586.788786] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15586.814274] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15591.245110] Lustre: server umount fs15246-MDT0000 complete [15591.909215] LustreError: 512686:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15597.029256] LustreError: 510840:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15597.641164] Lustre: server umount fs15246-OST0000 complete [15606.760444] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15606.768921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15606.786182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15612.388032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15612.397657] Lustre: Skipped 3 previous similar messages [15617.509827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15617.516935] Lustre: Skipped 1 previous similar message [15620.064140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15620.215518] Lustre: server umount lustre-OST0000 complete [15625.700264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15625.706912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15625.719509] Lustre: Skipped 1 previous similar message [15625.723060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15627.759425] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15629.564391] Lustre: server umount lustre-MDT0000 complete [15632.606440] LustreError: 517908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787808573 with bad export cookie 2430430671465894349 [15632.610613] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15632.618854] LustreError: 517908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15633.021214] Lustre: server umount lustre-MDT0001 complete [15642.325697] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [15644.463839] Key type lgssc unregistered [15644.739554] LNet: 518662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15644.748308] LNetError: 518662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15644.761819] LNet: Removed LNI 192.168.204.136@tcp [15645.624194] Key type .llcrypt unregistered [15645.626685] Key type ._llcrypt unregistered [15662.586303] Key type ._llcrypt registered [15662.589321] Key type .llcrypt registered [15662.668207] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [15663.523468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15663.573356] alg: No test for adler32 (adler32-zlib) [15664.616858] Lustre: Lustre: Build Version: 2.17.57_48_g018c4cb [15664.827113] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [15666.488185] Key type lgssc registered [15667.460878] Lustre: Echo OBD driver; http://www.lustre.org/ [15700.076955] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [15709.131790] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [15747.789447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15747.805753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15748.999644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15749.059946] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15749.226731] Lustre: lustre-MDT0000: new disk, initializing [15749.283459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15749.297110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15753.256917] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15764.271352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15764.389478] Lustre: 523130:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15764.459489] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15764.463466] Lustre: Skipped 1 previous similar message [15764.518985] Lustre: lustre-MDT0001: new disk, initializing [15764.608356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15764.651808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15764.671669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15769.062630] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15780.863311] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15787.462504] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15796.435441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15796.820806] Lustre: lustre-OST0000: new disk, initializing [15796.827745] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15796.840675] Lustre: 525100:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15796.984978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15804.117399] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15804.596378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15804.613118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15804.839704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15813.880610] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15817.779904] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15817.948581] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15823.073519] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15823.339609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15830.516827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15830.517439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15830.524395] LustreError: Skipped 1 previous similar message [15830.528462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15830.553124] Lustre: Skipped 1 previous similar message [15830.587279] Lustre: Skipped 1 previous similar message [15832.131676] Lustre: server umount lustre-OST0000 complete [15836.642724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15836.658693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15836.668670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15840.739523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15840.740347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15840.770500] Lustre: Skipped 1 previous similar message [15842.899118] Lustre: server umount lustre-MDT0000 complete [15846.717478] LustreError: 523121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787808787 with bad export cookie 14052908091908372441 [15846.737685] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15847.058647] Lustre: server umount lustre-MDT0001 complete [15875.082306] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [15886.153362] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [15921.985201] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15620 sec ========= 01:34:20 (1787808860) [15924.101959] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:34:22 (1787808862) === [15928.774560] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:34:26 (1787808866) === [15977.152192] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [15980.679041] Key type lgssc unregistered [15981.129786] LNet: 531750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15981.136394] LNetError: 531750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15981.161050] LNet: Removed LNI 192.168.204.136@tcp [15982.311288] Key type .llcrypt unregistered [15982.313523] Key type ._llcrypt unregistered