[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 476102566 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002384] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007287] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010130] LSM: Security Framework initializing [ 0.011053] Yama: becoming mindful. [ 0.012033] SELinux: Initializing. [ 0.013059] *** VALIDATE selinux *** [ 0.021577] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026592] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027118] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028123] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029111] *** VALIDATE tmpfs *** [ 0.030452] *** VALIDATE proc *** [ 0.031268] *** VALIDATE cgroup *** [ 0.033005] *** VALIDATE cgroup2 *** [ 0.034258] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036122] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038028] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.042058] debug: unmapping init [mem 0xffffffffba859000-0xffffffffba860fff] [ 0.045186] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046565] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047037] ... version: 2 [ 0.048010] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050014] ... value mask: 0000ffffffffffff [ 0.051016] ... max period: 00007fffffffffff [ 0.052013] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054285] rcu: Hierarchical SRCU implementation. [ 0.056376] smp: Bringing up secondary CPUs ... [ 0.057579] x86: Booting SMP configuration: [ 0.058027] .... node #0, CPUs: #1 #2 #3 [ 0.062710] smp: Brought up 1 node, 4 CPUs [ 0.064012] smpboot: Max logical packages: 1 [ 0.065021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143139] node 0 deferred pages initialised in 75ms [ 0.146087] devtmpfs: initialized [ 0.147341] x86/mm: Memory block size: 128MB [ 0.151000] gcov: version magic: 0x41383552 [ 0.155188] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157210] pinctrl core: initialized pinctrl subsystem [ 0.158112] [ 0.158428] ************************************************************* [ 0.159010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160010] ** ** [ 0.161011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162013] ** ** [ 0.163014] ** This means that this kernel is built to expose internal ** [ 0.164014] ** IOMMU data structures, which may compromise security on ** [ 0.165013] ** your system. ** [ 0.166013] ** ** [ 0.167013] ** If you see this message and you are not debugging the ** [ 0.168014] ** kernel, report this immediately to your vendor! ** [ 0.169011] ** ** [ 0.170008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171012] ************************************************************* [ 0.172544] NET: Registered protocol family 16 [ 0.173333] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.174051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.175054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.176320] cpuidle: using governor menu [ 0.178667] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.180350] PCI: Using configuration type 1 for base access [ 0.182093] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189149] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192077] cryptd: max_cpu_qlen set to 1000 [ 0.194255] ACPI: Added _OSI(Module Device) [ 0.196013] ACPI: Added _OSI(Processor Device) [ 0.198013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.200015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.204000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.211675] ACPI: Interpreter enabled [ 0.214062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.215010] ACPI: Using IOAPIC for interrupt routing [ 0.216735] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.220474] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.231387] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.233033] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.235016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.238072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.242345] acpiphp: Slot [2] registered [ 0.243124] acpiphp: Slot [5] registered [ 0.245100] acpiphp: Slot [6] registered [ 0.246088] acpiphp: Slot [7] registered [ 0.247152] acpiphp: Slot [8] registered [ 0.248118] acpiphp: Slot [9] registered [ 0.249092] acpiphp: Slot [10] registered [ 0.250101] acpiphp: Slot [3] registered [ 0.252140] acpiphp: Slot [4] registered [ 0.253091] acpiphp: Slot [11] registered [ 0.254101] acpiphp: Slot [12] registered [ 0.256112] acpiphp: Slot [13] registered [ 0.257085] acpiphp: Slot [14] registered [ 0.259096] acpiphp: Slot [15] registered [ 0.260082] acpiphp: Slot [16] registered [ 0.261188] acpiphp: Slot [17] registered [ 0.263086] acpiphp: Slot [18] registered [ 0.264136] acpiphp: Slot [19] registered [ 0.265096] acpiphp: Slot [20] registered [ 0.267079] acpiphp: Slot [21] registered [ 0.268070] acpiphp: Slot [22] registered [ 0.269092] acpiphp: Slot [23] registered [ 0.270125] acpiphp: Slot [24] registered [ 0.271076] acpiphp: Slot [25] registered [ 0.273073] acpiphp: Slot [26] registered [ 0.274128] acpiphp: Slot [27] registered [ 0.275066] acpiphp: Slot [28] registered [ 0.276106] acpiphp: Slot [29] registered [ 0.277071] acpiphp: Slot [30] registered [ 0.278067] acpiphp: Slot [31] registered [ 0.279082] PCI host bridge to bus 0000:00 [ 0.281015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.283017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.285020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.287016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.289022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.291026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.293149] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.296103] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.299341] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.309761] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.314674] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.317020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.319017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.322015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.324464] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.326881] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.328040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.331472] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.336015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.349962] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.353970] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.358799] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.367013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.374013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.389015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.399642] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.413015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.422015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.436013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.446013] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.452016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.459016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.479016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.490837] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.498014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.506014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.521014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.538106] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.547066] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.553016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.566015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.575674] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.586016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.598015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.624016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.639063] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.642379] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.645390] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.648447] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.652223] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.656094] iommu: Default domain type: Passthrough [ 0.657444] SCSI subsystem initialized [ 0.661235] ACPI: bus type USB registered [ 0.663093] usbcore: registered new interface driver usbfs [ 0.665088] usbcore: registered new interface driver hub [ 0.667085] usbcore: registered new device driver usb [ 0.669182] pps_core: LinuxPPS API ver. 1 registered [ 0.672012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.675060] PTP clock support registered [ 0.677085] EDAC MC: Ver: 3.0.0 [ 0.679468] PCI: Using ACPI for IRQ routing [ 0.681259] NetLabel: Initializing [ 0.683010] NetLabel: domain hash size = 128 [ 0.685011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.687077] NetLabel: unlabeled traffic allowed by default [ 0.690051] vgaarb: loaded [ 0.691321] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.693019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.704003] clocksource: Switched to clocksource kvm-clock [ 0.814510] VFS: Disk quotas dquot_6.6.0 [ 0.816378] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.819164] *** VALIDATE ramfs *** [ 0.820361] *** VALIDATE hugetlbfs *** [ 0.821945] pnp: PnP ACPI init [ 0.825070] pnp: PnP ACPI: found 6 devices [ 0.839712] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.842818] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.844820] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.846842] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.849102] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.851286] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.854253] NET: Registered protocol family 2 [ 0.856490] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.860877] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.864189] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.869516] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.872823] TCP: Hash tables configured (established 65536 bind 65536) [ 0.875513] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.878423] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.881115] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.883878] NET: Registered protocol family 1 [ 0.886255] RPC: Registered named UNIX socket transport module. [ 0.888293] RPC: Registered udp transport module. [ 0.889984] RPC: Registered tcp transport module. [ 0.891552] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.894064] NET: Registered protocol family 44 [ 0.895252] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.898158] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.899874] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.901566] PCI: CLS 0 bytes, default 64 [ 0.903480] Unpacking initramfs... [ 2.288530] debug: unmapping init [mem 0xffff97963cc54000-0xffff97963ffbffff] [ 2.292797] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.295346] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.298466] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.786222] Initialise system trusted keyrings [ 2.787958] Key type blacklist registered [ 2.789602] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.798098] zbud: loaded [ 2.800867] *** VALIDATE nfs *** [ 2.802215] *** VALIDATE nfs4 *** [ 2.803953] pstore: using deflate compression [ 2.807162] Platform Keyring initialized [ 2.908351] NET: Registered protocol family 38 [ 2.910068] Key type asymmetric registered [ 2.911244] Asymmetric key parser 'x509' registered [ 2.912503] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.915347] io scheduler mq-deadline registered [ 2.916736] io scheduler kyber registered [ 2.918265] io scheduler bfq registered [ 2.920121] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.922758] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.925563] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.928953] ACPI: Power Button [PWRF] [ 2.934260] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.942534] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.964495] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.978373] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.004607] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.033708] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.064637] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.069397] Non-volatile memory driver v1.3 [ 3.070812] Linux agpgart interface v0.103 [ 3.102619] virtio_blk virtio1: [vda] 146008 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.105936] vda: detected capacity change from 0 to 74756096 [ 3.122812] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.127874] vdb: detected capacity change from 0 to 1073741824 [ 3.142614] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.144957] vdc: detected capacity change from 0 to 2621440000 [ 3.157819] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.161184] vdd: detected capacity change from 0 to 2621440000 [ 3.177798] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.180158] vde: detected capacity change from 0 to 4294967296 [ 3.193479] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.196686] vdf: detected capacity change from 0 to 4294967296 [ 3.203632] libphy: Fixed MDIO Bus: probed [ 3.211907] usbcore: registered new interface driver usbserial_generic [ 3.213926] usbserial: USB Serial support registered for generic [ 3.216571] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.221255] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.223298] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.227218] mousedev: PS/2 mouse device common for all mice [ 3.230561] rtc_cmos 00:05: RTC can wake from S4 [ 3.234598] rtc_cmos 00:05: registered as rtc0 [ 3.235146] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.237121] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.243391] intel_pstate: CPU model not supported [ 3.246784] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.249186] hid: raw HID events driver (C) Jiri Kosina [ 3.252681] usbcore: registered new interface driver usbhid [ 3.254431] usbhid: USB HID core driver [ 3.256164] drop_monitor: Initializing network drop monitor service [ 3.256211] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.258450] Initializing XFRM netlink socket [ 3.258772] NET: Registered protocol family 10 [ 3.268536] Segment Routing with IPv6 [ 3.270211] NET: Registered protocol family 17 [ 3.272510] mpls_gso: MPLS GSO support [ 3.279170] RAS: Correctable Errors collector initialized. [ 3.281433] AVX version of gcm_enc/dec engaged. [ 3.283352] AES CTR mode by8 optimization enabled [ 3.357428] sched_clock: Marking stable (3357408538, 0)->(4273092293, -915683755) [ 3.362054] registered taskstats version 1 [ 3.364980] Loading compiled-in X.509 certificates [ 3.367173] zswap: loaded using pool lzo/zbud [ 3.391420] Key type big_key registered [ 3.404793] Key type encrypted registered [ 3.406784] ima: No TPM chip found, activating TPM-bypass! [ 3.408923] ima: Allocated hash algorithm: sha1 [ 3.410946] ima: No architecture policies found [ 3.412976] evm: Initialising EVM extended attributes: [ 3.415069] evm: security.selinux [ 3.416315] evm: security.ima [ 3.417344] evm: security.capability [ 3.418980] evm: HMAC attrs: 0x1 [ 3.421580] rtc_cmos 00:05: setting system clock to 2026-08-19 04:51:18 UTC (1787115078) [ 3.428522] debug: unmapping init [mem 0xffffffffbb803000-0xffffffffbb9fffff] [ 3.431235] debug: unmapping init [mem 0xffffffffba582000-0xffffffffba858fff] [ 3.441075] Write protecting the kernel read-only data: 28672k [ 3.444214] debug: unmapping init [mem 0xffffffffb8c03000-0xffffffffb8dfffff] [ 3.447038] debug: unmapping init [mem 0xffffffffb9514000-0xffffffffb95fffff] [ 3.479260] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.488589] systemd[1]: Detected virtualization kvm. [ 3.490556] systemd[1]: Detected architecture x86-64. [ 3.492542] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.518444] systemd[1]: No hostname configured. [ 3.520373] systemd[1]: Set hostname to . [ 3.522561] random: systemd: uninitialized urandom read (16 bytes read) [ 3.525213] systemd[1]: Initializing machine ID from random generator. [ 3.563989] random: ln: uninitialized urandom read (6 bytes read) [ 3.641977] random: systemd: uninitialized urandom read (16 bytes read) [ 3.646543] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.652557] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.657057] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.184544] device-mapper: uevent: version 1.0.3 [ 4.186531] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.806542] random: fast init done [ 4.812806] virtio_net virtio0 ens2: renamed from eth0 [ 4.861722] scsi host0: ata_piix [ 4.870657] scsi host1: ata_piix [ 4.872364] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.875412] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.121223] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.923737] random: crng init done [ 9.925094] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.127661] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.190612] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.470414] SELinux: Disabled at runtime. [ 11.531915] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.538575] systemd[1]: Detected virtualization kvm. [ 11.540231] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.025388] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.029313] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.034554] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.039248] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.043305] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.051336] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.059941] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ 12.218703] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.472348] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.804741] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.805966] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.982035] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.998780] EDAC sbridge: Ver: 1.1.2 [ 14.476461] Key type dns_resolver registered [ 14.773082] NFS: Registering the id_resolver key type [ 14.775401] Key type id_resolver registered [ 14.777414] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg344-server login: [ 41.078922] libcfs: loading out-of-tree module taints kernel. [ 41.099600] Key type ._llcrypt registered [ 41.101395] Key type .llcrypt registered [ 41.151408] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 49.984146] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 50.580685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.588325] alg: No test for adler32 (adler32-zlib) [ 51.627529] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 51.975773] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 53.607224] Key type lgssc registered [ 55.575963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 76.247643] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 124.819840] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 136.952437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 136.990926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 138.472480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 138.504411] hrtimer: interrupt took 8380126 ns [ 138.532677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 138.697261] Lustre: lustre-MDT0000: new disk, initializing [ 138.835729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 138.860317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 143.242195] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 156.918532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 156.995562] Lustre: 6483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 157.020162] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 157.023977] Lustre: Skipped 1 previous similar message [ 157.096964] Lustre: lustre-MDT0001: new disk, initializing [ 157.170808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 157.202083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 157.209903] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 161.732387] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 166.334898] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 175.662689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 176.082259] Lustre: lustre-OST0000: new disk, initializing [ 176.087126] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 176.093624] Lustre: 8425:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 176.179503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 182.330727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 182.339289] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 182.440570] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 182.630717] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.340186] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 202.623921] Lustre: lustre-OST0001: new disk, initializing [ 202.643907] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 202.657444] Lustre: 9497:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 202.788581] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 210.353867] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.627583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 212.643598] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 212.705040] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 222.516534] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 233.380340] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 240.280964] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing check_logdir /tmp/testlogs/ [ 245.962996] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing yml_node [ 250.589961] Lustre: DEBUG MARKER: Client: 2.17.56.4 [ 253.397979] Lustre: DEBUG MARKER: MDS: 2.17.56.4 [ 256.153118] Lustre: DEBUG MARKER: OSS: 2.17.56.4 [ 257.874965] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 19 00:55:32 EDT 2026 [ 275.190481] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 277.316326] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 300.004761] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.015370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 300.026909] Lustre: Skipped 3 previous similar messages [ 300.041769] Lustre: Skipped 3 previous similar messages [ 302.532590] Lustre: server umount lustre-MDT0000 complete [ 307.015324] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787115382 with bad export cookie 322258018143888712 [ 307.023929] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 307.024957] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 310.246079] LustreError: 9308:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.255235] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.281312] LustreError: 9308:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 310.335947] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 310.361251] Lustre: Skipped 1 previous similar message [ 312.288482] LustreError: 9308:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 312.291571] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 312.330888] Lustre: Skipped 1 previous similar message [ 313.318180] Lustre: server umount lustre-MDT0001 complete [ 318.236862] Lustre: server umount lustre-OST0000 complete [ 323.262306] Lustre: server umount lustre-OST0001 complete [ 334.212211] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 350.997188] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 391.785180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 392.255855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 392.367703] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 392.521311] Lustre: lustre-MDT0000: new disk, initializing [ 392.665586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 392.703598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 399.179476] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 411.611817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.731224] Lustre: 15073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 411.794873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 411.805624] Lustre: Skipped 1 previous similar message [ 411.892708] Lustre: lustre-MDT0001: new disk, initializing [ 412.029694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 412.075209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 412.096455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 417.399349] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 427.177439] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 432.915277] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 438.890257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 439.106935] Lustre: lustre-OST0000: new disk, initializing [ 439.110100] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 439.115598] Lustre: 17045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 439.197562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 440.324420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 440.331224] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 440.433197] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 444.644126] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 453.933850] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 458.298345] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 458.550655] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 462.452976] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 462.630725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 465.893543] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.895690] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 465.906270] Lustre: Skipped 1 previous similar message [ 465.913691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 470.484673] Lustre: server umount lustre-OST0000 complete [ 476.136926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 476.169725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 476.174410] Lustre: Skipped 2 previous similar messages [ 480.262416] Lustre: server umount lustre-MDT0000 complete [ 481.248457] LustreError: 15082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 481.267802] LustreError: 15082:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 483.700110] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787115558 with bad export cookie 322258018143890245 [ 483.702550] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 483.719217] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 484.156335] Lustre: server umount lustre-MDT0001 complete [ 489.606990] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 00:59:23 (1787115563) [ 493.110693] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 495.587643] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 00:59:29 (1787115569) [ 504.623353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 505.492959] LustreError: 19455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 505.686685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 510.584605] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 510.945409] LustreError: 19456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.720960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 521.057483] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 528.087480] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 534.151653] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 540.775811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 547.673625] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 555.502995] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 565.730996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 565.732728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 565.763017] Lustre: Skipped 2 previous similar messages [ 565.772368] Lustre: Skipped 1 previous similar message [ 569.246432] Lustre: server umount lustre-OST0000 complete [ 574.435746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 574.458167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 580.634765] Lustre: server umount lustre-MDT0000 complete [ 581.099382] LustreError: 21896:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 581.140657] LustreError: 21896:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 584.743378] LustreError: 19436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787115659 with bad export cookie 322258018143891127 [ 584.744554] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 584.761487] LustreError: 19436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 585.205655] Lustre: server umount lustre-MDT0001 complete [ 597.636301] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 600.766734] Key type lgssc unregistered [ 601.031433] LNet: 23000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 601.036379] LNetError: 23000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 601.047900] LNet: Removed LNI 192.168.203.144@tcp [ 602.034164] Key type .llcrypt unregistered [ 602.037591] Key type ._llcrypt unregistered [ 629.057469] Key type ._llcrypt registered [ 629.062632] Key type .llcrypt registered [ 629.170394] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 630.527461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 630.557058] alg: No test for adler32 (adler32-zlib) [ 631.771404] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 632.006899] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 633.743235] Key type lgssc registered [ 634.808762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 644.045366] loop: module loaded [ 644.148557] loop0: detected capacity change from 0 to 204800000 [ 644.416430] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 644.443753] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 644.742470] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 650.437980] loop0: detected capacity change from 0 to 204800000 [ 650.605622] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 650.626982] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 650.640832] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 655.494286] loop0: detected capacity change from 0 to 204800000 [ 660.492903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 660.542109] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 661.782749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 661.832702] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 661.888387] Lustre: lustre-MDT0000: new disk, initializing [ 661.947765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 661.963572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 666.132846] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 675.331399] loop1: detected capacity change from 0 to 204800000 [ 681.362759] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 681.620945] Lustre: lustre-OST07c6: new disk, initializing [ 681.627207] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 681.633559] Lustre: Skipped 1 previous similar message [ 681.638440] Lustre: 26187:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 681.717914] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 687.784145] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 691.797225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 691.804895] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 691.843547] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 697.741034] Lustre: 26703:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 697.753461] Lustre: 26703:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 697.760405] Lustre: 26703:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 697.766586] Lustre: 26703:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 697.772573] Lustre: 26703:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 697.780880] Lustre: 26703:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 714.532322] Lustre: 25172:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 714.539402] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 714.543319] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 714.548724] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 714.553802] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 714.557799] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 719.394291] Lustre: 25170:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 719.410100] Lustre: 25170:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 719.423183] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 719.427777] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.431467] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 719.435760] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.441784] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 719.446872] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.452844] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 719.458917] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.464134] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 719.469716] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.739540] Lustre: 25172:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 722.750762] Lustre: 25172:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 722.756989] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 722.760498] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.765492] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 722.770529] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.773724] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 722.779101] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.784182] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 722.788858] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.793836] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 722.799260] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.490490] Lustre: 25170:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 727.503033] Lustre: 25170:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 727.518132] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 727.527272] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.533687] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 727.541830] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.548352] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 727.556837] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.568054] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 727.581245] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.586309] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 727.592234] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 738.419355] Lustre: 26703:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 738.434238] Lustre: 26703:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 738.444289] Lustre: 26703:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 738.450300] Lustre: 26703:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 738.465070] Lustre: 26703:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 738.478264] Lustre: 26703:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 738.488424] Lustre: 26703:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 738.496651] Lustre: 26703:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 738.505210] Lustre: 26703:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 738.516460] Lustre: 26703:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 738.523975] Lustre: 26703:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 738.531552] Lustre: 26703:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 788.963380] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 788.967203] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 788.978678] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 791.526484] Lustre: server umount lustre-OST07c6 complete [ 799.982168] Lustre: server umount lustre-MDT0000 complete [ 814.627533] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 817.416954] Key type lgssc unregistered [ 817.742728] LNet: 28750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.759046] LNetError: 28750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.803879] LNet: Removed LNI 192.168.203.144@tcp [ 818.697313] Key type .llcrypt unregistered [ 818.703993] Key type ._llcrypt unregistered [ 840.086134] Key type ._llcrypt registered [ 840.088160] Key type .llcrypt registered [ 840.189638] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 841.140939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 841.281096] alg: No test for adler32 (adler32-zlib) [ 842.261205] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 842.535047] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 844.191154] Key type lgssc registered [ 845.258991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 852.674671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 852.702757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 854.156775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 858.943826] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 864.741727] LustreError: 30021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 864.773307] LustreError: 30021:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 867.606046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 867.974662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 872.072069] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 878.986801] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 884.617406] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 891.956384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 892.306314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 898.071287] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 899.515322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 905.513215] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 924.128368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 924.137174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 924.157363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 925.154065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 925.161386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 925.176056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 926.743582] Lustre: server umount lustre-OST0000 complete [ 930.273643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 930.275454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 930.288973] Lustre: Skipped 1 previous similar message [ 930.296251] Lustre: Skipped 1 previous similar message [ 935.394302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 935.405167] Lustre: Skipped 2 previous similar messages [ 940.513554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 940.521562] Lustre: Skipped 1 previous similar message [ 944.095490] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 944.277815] Lustre: server umount lustre-MDT0000 complete [ 945.634026] LustreError: 30749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 945.667172] LustreError: 30749:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 948.209001] LustreError: 30003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116023 with bad export cookie 4905705746657937987 [ 948.216946] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 948.224346] LustreError: 30003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 948.722337] Lustre: server umount lustre-MDT0001 complete [ 960.178430] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 962.925421] Key type lgssc unregistered [ 963.276698] LNet: 33759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.291286] LNetError: 33759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.317906] LNet: Removed LNI 192.168.203.144@tcp [ 964.158476] Key type .llcrypt unregistered [ 964.163441] Key type ._llcrypt unregistered [ 984.565681] Key type ._llcrypt registered [ 984.568270] Key type .llcrypt registered [ 984.660404] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 985.906171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 985.985227] alg: No test for adler32 (adler32-zlib) [ 987.050552] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 987.309720] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 988.976110] Key type lgssc registered [ 990.096569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 996.828358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 996.854767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 998.240118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1002.403113] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1008.622218] LustreError: 35033:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1008.638867] LustreError: 35033:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1009.385274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1009.634448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1013.756865] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1020.826418] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1026.788312] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1033.136442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1033.507593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1038.909580] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1043.763976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1045.684978] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1059.301556] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1059.303412] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1059.312921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1059.326636] Lustre: Skipped 1 previous similar message [ 1062.169450] Lustre: server umount lustre-OST0000 complete [ 1065.964728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1065.978479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1065.992285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1065.998609] Lustre: Skipped 1 previous similar message [ 1069.539104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1069.540546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1071.750179] Lustre: server umount lustre-MDT0000 complete [ 1074.664554] LustreError: 35033:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1074.688044] LustreError: 35033:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1075.433032] LustreError: 35012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116150 with bad export cookie 15776013314936815994 [ 1075.436302] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1075.458086] LustreError: 35012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1075.909473] Lustre: server umount lustre-MDT0001 complete [ 1087.696190] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1090.595752] Key type lgssc unregistered [ 1091.004434] LNet: 38566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1091.017914] LNetError: 38566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1092.068736] LNet: Removed LNI 192.168.203.144@tcp [ 1092.830459] Key type .llcrypt unregistered [ 1092.834927] Key type ._llcrypt unregistered [ 1118.120513] Key type ._llcrypt registered [ 1118.123935] Key type .llcrypt registered [ 1118.289255] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1119.354276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1119.541109] alg: No test for adler32 (adler32-zlib) [ 1120.555775] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 1120.832853] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1122.559738] Key type lgssc registered [ 1123.791878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1130.709156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1130.744591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1132.149227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1137.367868] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1142.770394] LustreError: 39984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1142.799710] LustreError: 39984:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1146.248462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1146.592552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1151.368126] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1159.450135] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1166.552435] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1173.876678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1174.322510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1177.395329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1181.083594] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1189.058319] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1197.027974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1197.031647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1197.047874] Lustre: Skipped 1 previous similar message [ 1197.066117] Lustre: Skipped 2 previous similar messages [ 1202.145701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1202.158571] Lustre: Skipped 3 previous similar messages [ 1202.404126] Lustre: server umount lustre-MDT0000 complete [ 1205.957210] LustreError: 39965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116281 with bad export cookie 8217171878441206959 [ 1205.964319] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1205.981464] LustreError: 39965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1207.265944] LustreError: 39979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1207.272274] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1207.323994] Lustre: Skipped 1 previous similar message [ 1207.334417] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1207.339133] Lustre: Skipped 1 previous similar message [ 1210.276138] LustreError: 39980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1210.276442] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1210.295169] LustreError: 39980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1210.308826] Lustre: Skipped 1 previous similar message [ 1212.475681] Lustre: server umount lustre-MDT0001 complete [ 1233.788622] Lustre: server umount lustre-OST0000 complete [ 1249.678400] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1252.534731] Key type lgssc unregistered [ 1252.857443] LNet: 43845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1252.865712] LNetError: 43845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1252.895355] LNet: Removed LNI 192.168.203.144@tcp [ 1253.768447] Key type .llcrypt unregistered [ 1253.775118] Key type ._llcrypt unregistered [ 1275.065949] Key type ._llcrypt registered [ 1275.068377] Key type .llcrypt registered [ 1275.162441] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1276.109193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1276.272116] alg: No test for adler32 (adler32-zlib) [ 1277.248908] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 1277.437192] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1279.151166] Key type lgssc registered [ 1280.388385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1288.654726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1288.671680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1290.133990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1294.221923] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1300.452646] LustreError: 45261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1300.476459] LustreError: 45261:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1302.030036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1302.438614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1307.236658] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1314.799137] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1320.399423] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1326.708549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1327.055981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1333.306926] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1335.288798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1342.764962] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1349.745585] Lustre: lustre-OST0000: Not available for connect from 192.168.203.44@tcp (stopping) [ 1350.625356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1350.658502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1354.824777] Lustre: lustre-OST0000: Not available for connect from 192.168.203.44@tcp (stopping) [ 1354.842514] Lustre: Skipped 1 previous similar message [ 1355.120907] Lustre: server umount lustre-OST0000 complete [ 1374.176568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1374.186437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1374.197440] Lustre: Skipped 1 previous similar message [ 1374.206362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1375.202564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1378.907143] Lustre: server umount lustre-MDT0000 complete [ 1380.322072] LustreError: 45261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1380.335320] LustreError: 45261:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1381.926563] LustreError: 45242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116457 with bad export cookie 3674595377348143274 [ 1381.931643] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1381.932694] LustreError: 45242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1388.349670] Lustre: server umount lustre-MDT0001 complete [ 1400.007727] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1402.564235] Key type lgssc unregistered [ 1402.879516] LNet: 48977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1402.894827] LNetError: 48977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1402.920479] LNet: Removed LNI 192.168.203.144@tcp [ 1403.819491] Key type .llcrypt unregistered [ 1403.822422] Key type ._llcrypt unregistered [ 1428.761987] Key type ._llcrypt registered [ 1428.764154] Key type .llcrypt registered [ 1428.916340] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1430.153200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1430.186839] alg: No test for adler32 (adler32-zlib) [ 1431.264168] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 1431.480517] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1433.151185] Key type lgssc registered [ 1434.016810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1441.501179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1441.525813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1443.005507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1447.519489] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1453.537805] LustreError: 50391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1453.549138] LustreError: 50391:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1455.825947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1456.238101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1460.368567] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1467.607130] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1473.501366] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1480.054102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1480.522518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1485.787560] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1488.698480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1493.508958] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1498.517936] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1501.141989] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1513.983458] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-19 1h16m26s [ 1516.115695] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1519.591793] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1519.596435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1519.617672] Lustre: Skipped 2 previous similar messages [ 1519.647410] Lustre: Skipped 2 previous similar messages [ 1520.675816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1520.689224] Lustre: Skipped 2 previous similar messages [ 1522.318539] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.44@tcp (stopping) [ 1522.338617] Lustre: Skipped 1 previous similar message [ 1524.908045] Lustre: server umount lustre-MDT0000 complete [ 1526.756441] LustreError: 50392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1526.778442] LustreError: 50392:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1528.858229] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1532.899954] LustreError: 50391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1532.923059] LustreError: 50391:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1535.448432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1535.568442] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1535.843247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1535.940633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1536.884847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1540.253686] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1543.464184] Lustre: DEBUG MARKER: Wait for df (36693) ... [ 1545.216076] Lustre: DEBUG MARKER: done [ 1551.329080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1551.341936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1551.359077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1552.362045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1555.852280] Lustre: server umount lustre-OST0000 complete [ 1562.603645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1562.604125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1562.662435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1562.665016] Lustre: Skipped 2 previous similar messages [ 1573.856678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1573.871546] Lustre: Skipped 4 previous similar messages [ 1574.368476] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1574.520669] Lustre: server umount lustre-MDT0000 complete [ 1578.352765] LustreError: 50374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116653 with bad export cookie 13614991332395537492 [ 1578.356631] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1578.363644] LustreError: 50374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1578.982808] Lustre: server umount lustre-MDT0001 complete [ 1591.822880] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1595.030896] Key type lgssc unregistered [ 1595.455569] LNet: 55324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1595.461587] LNetError: 55324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1595.477277] LNet: Removed LNI 192.168.203.144@tcp [ 1596.595110] Key type .llcrypt unregistered [ 1596.599932] Key type ._llcrypt unregistered [ 1636.931783] Key type ._llcrypt registered [ 1636.933963] Key type .llcrypt registered [ 1637.034223] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1637.978461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1638.052480] alg: No test for adler32 (adler32-zlib) [ 1639.091718] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 1639.349473] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1640.991156] Key type lgssc registered [ 1642.240884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1650.286321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1650.307451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1651.651671] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1651.682388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1651.847402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1656.396743] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1665.232157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1665.321820] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1665.356301] Lustre: 57538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1665.681717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1670.314588] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1679.294649] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1685.701729] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1693.467541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1693.770846] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1694.109476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1700.530831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1701.826929] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1710.896840] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1718.896035] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-19 1h19m52s [ 1720.675765] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1729.188903] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1731.940184] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1732.040312] LustreError: 57542:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1752.187291] Lustre: lustre-MDT0000: Client 5f083485-ce8b-4dd9-819f-fc891b737058 (at 192.168.203.44@tcp) reconnecting [ 1772.079125] LustreError: 57542:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1774.583839] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1776.948640] Lustre: DEBUG MARKER: done [ 1787.361123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1787.368341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1787.379554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1787.872945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1787.885472] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1787.908537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1789.326205] Lustre: server umount lustre-OST0000 complete [ 1794.017921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1794.030952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1794.048724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1799.146473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1799.153112] Lustre: Skipped 4 previous similar messages [ 1799.569753] Lustre: server umount lustre-MDT0000 complete [ 1803.278359] LustreError: 59406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116878 with bad export cookie 7587310539376820715 [ 1803.290109] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1803.620740] Lustre: server umount lustre-MDT0001 complete [ 1814.684113] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1817.652520] Key type lgssc unregistered [ 1817.979881] LNet: 61866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1817.986259] LNetError: 61866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1818.004667] LNet: Removed LNI 192.168.203.144@tcp [ 1818.724329] Key type .llcrypt unregistered [ 1818.727256] Key type ._llcrypt unregistered [ 1872.346040] Key type ._llcrypt registered [ 1872.351239] Key type .llcrypt registered [ 1872.484124] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1873.941813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1874.013858] alg: No test for adler32 (adler32-zlib) [ 1875.028032] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 1875.282405] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1877.039901] Key type lgssc registered [ 1878.195931] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1881.339551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1881.354752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1882.534791] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1882.553317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1882.672807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1883.664107] Lustre: Failing over lustre-MDT0000 [ 1883.912737] Lustre: server umount lustre-MDT0000 complete [ 1891.504074] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:22:45 (1787116965) [ 1899.228654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1899.995677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1906.725539] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1918.640733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1918.997818] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1919.058604] Lustre: 64665:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1919.547448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1924.875908] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1933.797938] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1940.345530] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1947.153283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1947.296451] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1947.502862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1953.499796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1954.832692] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1963.551347] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1969.483465] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1979.365123] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1979.385763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1979.390589] Lustre: Skipped 2 previous similar messages [ 1979.402134] Lustre: Skipped 2 previous similar messages [ 1982.600451] Lustre: server umount lustre-MDT0000 complete [ 1985.770456] LustreError: 65403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117060 with bad export cookie 16868513701982967241 [ 1985.773911] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1985.796108] LustreError: 65403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1986.242743] Lustre: server umount lustre-MDT0001 complete [ 1989.209446] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1994.354119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2003.536242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2004.959799] Lustre: 63589:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787117064/real 1787117064] req@00000000b6049abd x1873927943006208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787117080 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2004.980365] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.985366] Lustre: 63590:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787117065/real 1787117065] req@00000000d8468a11 x1873927943006336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787117081 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2010.079134] Lustre: 63591:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787117069/real 1787117069] req@0000000005077858 x1873927943006464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787117085 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2010.143634] LustreError: 63587:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001a7e84c7 x1873927943007232/t0(0) o250->MGC192.168.203.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2010.489250] LustreError: 66516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2010.515660] LustreError: 66516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2010.608957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2010.756262] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2014.626876] LustreError: 66517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2015.006725] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2015.653377] LustreError: 66516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2015.671890] LustreError: 66516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2016.671456] Lustre: 63588:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787117075/real 1787117075] req@000000004646a9d6 x1873927943006848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787117091 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2016.705697] Lustre: 63588:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2020.840058] LustreError: 68436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2020.861225] LustreError: 68436:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2023.918250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2024.163635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2027.832171] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2029.556293] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2034.730308] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2034.858541] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2041.304937] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2050.018673] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2050.019294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2050.033971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2055.140318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2055.146894] Lustre: Skipped 2 previous similar messages [ 2055.511807] Lustre: server umount lustre-MDT0000 complete [ 2059.315641] LustreError: 65403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117134 with bad export cookie 16868513701982969516 [ 2059.317981] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2059.331501] LustreError: 65403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2059.841559] Lustre: server umount lustre-MDT0001 complete [ 2066.650835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2076.640108] Lustre: 63588:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787117135/real 1787117135] req@0000000056835715 x1873927943032192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787117151 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2076.679840] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2076.713988] Lustre: Skipped 2 previous similar messages [ 2077.479985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2084.325198] LustreError: 66516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2084.366152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2084.424789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2085.408098] Lustre: 63588:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787117144/real 1787117144] req@00000000a09968c3 x1873927943032704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787117160 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2085.457320] Lustre: 63588:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2089.822141] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2096.619399] LustreError: 71121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2096.654838] LustreError: 71121:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2099.942172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2105.591451] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2105.722791] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2114.320256] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2120.807790] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2131.427385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2131.441189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2131.444147] Lustre: Skipped 1 previous similar message [ 2134.442804] Lustre: server umount lustre-MDT0000 complete [ 2136.548264] LustreError: 71121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.578557] LustreError: 71121:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2137.962837] LustreError: 64657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117213 with bad export cookie 16868513701982970895 [ 2137.964892] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2137.981164] LustreError: 64657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2138.292991] Lustre: server umount lustre-MDT0001 complete [ 2144.517509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2148.499825] Lustre: DEBUG MARKER: files compared the same [ 2160.942662] Lustre: server umount lustre-OST0000 complete [ 2176.988347] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2180.663327] Key type lgssc unregistered [ 2181.003773] LNet: 74396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2181.022346] LNetError: 74396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2181.053289] LNet: Removed LNI 192.168.203.144@tcp [ 2182.357167] Key type .llcrypt unregistered [ 2182.360446] Key type ._llcrypt unregistered [ 2209.556905] Key type ._llcrypt registered [ 2209.560237] Key type .llcrypt registered [ 2209.697684] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2211.245503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2211.287928] alg: No test for adler32 (adler32-zlib) [ 2212.333375] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 2212.595074] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2214.361627] Key type lgssc registered [ 2216.506523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2224.877669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2224.919139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2226.420455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2231.746806] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug +malloc all [ 2236.895618] LustreError: 75815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2236.921890] LustreError: 75815:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2241.344966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2241.580508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2245.359475] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug +malloc all [ 2254.427890] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2261.034716] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2269.376273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2269.791051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2271.860309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2276.469373] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug +malloc all [ 2284.749780] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2296.808607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2296.816336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2296.830662] Lustre: Skipped 1 previous similar message [ 2296.832071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2296.839108] Lustre: Skipped 1 previous similar message [ 2299.745707] Lustre: server umount lustre-OST0000 complete [ 2307.041206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2307.043116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2307.061170] Lustre: Skipped 1 previous similar message [ 2307.071846] Lustre: Skipped 1 previous similar message [ 2309.787769] Lustre: server umount lustre-MDT0000 complete [ 2312.162356] LustreError: 75811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2312.173872] LustreError: 75811:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2314.255948] LustreError: 75797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117389 with bad export cookie 9031362369574202586 [ 2314.264741] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2314.296231] LustreError: 75797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2314.835684] Lustre: server umount lustre-MDT0001 complete [ 2330.269301] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2333.832922] Key type lgssc unregistered [ 2334.246767] LNet: 79350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2334.251211] LNetError: 79350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2334.290168] LNet: Removed LNI 192.168.203.144@tcp [ 2335.211656] Key type .llcrypt unregistered [ 2335.213898] Key type ._llcrypt unregistered [ 2360.964581] Key type ._llcrypt registered [ 2360.967510] Key type .llcrypt registered [ 2361.124532] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2362.476570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2362.530148] alg: No test for adler32 (adler32-zlib) [ 2363.686899] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 2363.975411] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2365.671589] Key type lgssc registered [ 2366.914153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2375.174403] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2375.205800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2390.815462] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2395.938327] LustreError: 80745:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.144@tcp: failed processing log, type 4: rc = -110 [ 2422.687660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2429.729506] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2437.470942] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2447.780815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2447.891589] Lustre: *** cfs_fail_loc=706, val=0*** [ 2448.178808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2448.258847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2453.479918] LustreError: 80771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2453.497474] LustreError: 80771:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2454.256497] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2457.580132] LustreError: 80770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2457.626193] LustreError: 80770:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2462.689316] LustreError: 81831:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2462.713727] LustreError: 81831:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2463.362123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2463.816077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2469.488259] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2476.559355] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2482.745667] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2489.313600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2489.322218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2489.327843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2489.344354] Lustre: Skipped 1 previous similar message [ 2491.251269] Lustre: server umount lustre-OST0000 complete [ 2499.561954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2499.562659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2499.573586] LustreError: Skipped 1 previous similar message [ 2499.574393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2499.596896] Lustre: Skipped 1 previous similar message [ 2499.617865] Lustre: Skipped 2 previous similar messages [ 2501.724201] Lustre: server umount lustre-MDT0000 complete [ 2504.672327] LustreError: 81830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2504.696680] LustreError: 81830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2506.271170] LustreError: 80751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117581 with bad export cookie 4740859999564925349 [ 2506.284380] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2506.292840] LustreError: 80751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2506.614950] Lustre: server umount lustre-MDT0001 complete [ 2518.427521] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2521.299568] Key type lgssc unregistered [ 2521.659657] LNet: 84360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2521.669405] LNetError: 84360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2521.700449] LNet: Removed LNI 192.168.203.144@tcp [ 2522.764500] Key type .llcrypt unregistered [ 2522.767810] Key type ._llcrypt unregistered [ 2548.482774] Key type ._llcrypt registered [ 2548.485159] Key type .llcrypt registered [ 2548.613160] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2549.449532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2549.469795] alg: No test for adler32 (adler32-zlib) [ 2550.743673] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 2551.057398] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2552.807210] Key type lgssc registered [ 2554.302696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2563.382334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2563.428793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2569.949915] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2578.023196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2578.660530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2583.412057] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2584.036725] LustreError: 86465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.056386] LustreError: 86465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2588.130795] LustreError: 86469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2588.170204] LustreError: 86469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2592.863806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2593.251140] LustreError: 86468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2593.267451] LustreError: 86468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2593.306717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2598.372654] LustreError: 86469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.389394] LustreError: 86469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2599.751831] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2603.491151] LustreError: 86465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2603.535444] LustreError: 86465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2609.144992] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2609.154266] Lustre: MGS: Client c2bed0a2-77f1-4c92-98d6-af50ea4c6894 (at 0@lo) reconnecting [ 2609.166579] LustreError: 86479:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2609.419777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2609.533868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2614.599587] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2630.143211] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2630.158997] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2630.163066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2630.185517] Lustre: Skipped 1 previous similar message [ 2634.960375] Lustre: server umount lustre-OST0000 complete [ 2640.362158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2640.364430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2640.387273] Lustre: Skipped 1 previous similar message [ 2640.407651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2640.415396] Lustre: Skipped 1 previous similar message [ 2645.218495] Lustre: server umount lustre-MDT0000 complete [ 2645.472960] LustreError: 88270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2645.506673] LustreError: 88270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2651.348968] Lustre: server umount lustre-MDT0001 complete [ 2656.361392] Lustre: server umount MGS complete [ 2670.519872] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:35:44 (1787117744) [ 2703.454737] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 2716.490935] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2756.452219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2762.651673] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2775.060793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2775.321582] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2775.369777] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2775.463349] Lustre: lustre-MDT0001: new disk, initializing [ 2775.574531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2775.620147] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2775.630497] Lustre: 93637:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2775.644095] LustreError: 93637:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2780.045483] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2791.221728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2791.524661] Lustre: lustre-OST0000: new disk, initializing [ 2791.532489] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2791.543110] Lustre: 94405:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2791.655503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2798.721028] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2801.185237] Lustre: 94830:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000a2a667ca x1873928651842304/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787117892 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2808.793813] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2815.633558] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2815.646746] Lustre: MGS: Client 16ab82de-f627-4095-9ed6-33bf61ed4fc8 (at 0@lo) reconnecting [ 2815.679264] Lustre: 92904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2815.698438] LustreError: 93623:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2815.710376] LustreError: 93623:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2815.732333] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2815.828518] Lustre: lustre-MDT0000: new disk, initializing [ 2815.990728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2816.032975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2816.067550] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2817.122446] Lustre: 95474:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000001d3d65ea x1873928651848960/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787117908 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2820.924984] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2822.242593] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2826.307698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2826.324713] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2826.374953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2852.320614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2852.327941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2852.339302] Lustre: Skipped 1 previous similar message [ 2852.343698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2854.368383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2854.373458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2854.382534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2857.497106] Lustre: server umount lustre-OST0000 complete [ 2866.164123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2866.164812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2866.182736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2866.182743] Lustre: Skipped 2 previous similar messages [ 2868.517891] Lustre: server umount lustre-MDT0000 complete [ 2871.265882] LustreError: 93609:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2871.277539] LustreError: 93609:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2879.449679] Lustre: server umount lustre-MDT0001 complete [ 2883.600918] Lustre: server umount MGS complete [ 2897.442513] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 01:39:31 (1787117971) [ 2917.396916] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2920.825471] Key type lgssc unregistered [ 2921.185825] LNet: 97976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2921.198643] LNetError: 97976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2921.234594] LNet: Removed LNI 192.168.203.144@tcp [ 2922.181788] Key type .llcrypt unregistered [ 2922.186376] Key type ._llcrypt unregistered [ 2941.444107] Key type ._llcrypt registered [ 2941.447862] Key type .llcrypt registered [ 2941.570834] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2943.112492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2943.175411] alg: No test for adler32 (adler32-zlib) [ 2944.349140] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 2944.647252] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2946.327346] Key type lgssc registered [ 2947.730946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2956.080413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2956.113312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2957.581818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2962.905312] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2967.015532] LustreError: 99200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2967.047432] LustreError: 99200:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2972.129075] LustreError: 99201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2972.646989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2973.259698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2978.372592] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2987.674564] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2994.399885] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3004.839299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3005.609551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3014.183702] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3024.492051] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3027.134823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3042.806068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3042.815762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3042.838706] Lustre: Skipped 1 previous similar message [ 3042.839329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3042.872080] Lustre: Skipped 1 previous similar message [ 3046.269982] Lustre: server umount lustre-OST0000 complete [ 3053.025074] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3053.027408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3053.051083] Lustre: Skipped 1 previous similar message [ 3058.148141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3058.156824] Lustre: Skipped 2 previous similar messages [ 3062.245342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3062.257831] Lustre: Skipped 1 previous similar message [ 3065.311138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3065.568173] Lustre: server umount lustre-MDT0000 complete [ 3067.362671] LustreError: 100173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3067.405112] LustreError: 100173:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3070.053040] LustreError: 99181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118145 with bad export cookie 17204860071684869661 [ 3070.076385] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3070.829503] Lustre: server umount lustre-MDT0001 complete [ 3084.518728] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 3087.566225] Key type lgssc unregistered [ 3087.968654] LNet: 102756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3087.975246] LNetError: 102756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3087.995704] LNet: Removed LNI 192.168.203.144@tcp [ 3089.047339] Key type .llcrypt unregistered [ 3089.053159] Key type ._llcrypt unregistered [ 3109.826787] Key type ._llcrypt registered [ 3109.846995] Key type .llcrypt registered [ 3110.073876] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3111.832671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3112.031744] alg: No test for adler32 (adler32-zlib) [ 3113.253318] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 3113.759960] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 3115.527225] Key type lgssc registered [ 3117.081175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3125.142308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3125.154318] LustreError: 103833:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3125.198742] LustreError: 103840:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3125.204921] LustreError: 103833:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4957 [ 3125.217969] LustreError: 103833:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3125.225308] LustreError: 103833:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3125.229718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3126.970755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3135.775223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3136.156609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3141.965706] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3148.468693] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing lsmod [ 3151.978256] LustreError: 105043:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3152.008617] LustreError: 105048:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3152.015760] LustreError: 105043:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4970 [ 3152.029959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3152.032028] LustreError: 105043:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3152.032081] LustreError: 105043:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3152.485967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3153.528867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3156.960621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3156.971291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3156.993343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3159.009164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3159.016323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3159.028645] Lustre: Skipped 1 previous similar message [ 3159.068550] Lustre: Skipped 2 previous similar messages [ 3161.064981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3161.092081] Lustre: Skipped 2 previous similar messages [ 3162.029296] Lustre: server umount lustre-MDT0000 complete [ 3165.975735] LustreError: 104317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118241 with bad export cookie 6333586130906016518 [ 3165.986338] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3165.994512] LustreError: 104317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3166.177085] LustreError: 103873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3166.181282] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3166.195401] LustreError: 103873:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3166.225476] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3167.264834] LustreError: 104310:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d589f0d1 x1873929241123328/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3167.476747] Lustre: server umount lustre-MDT0001 complete [ 3171.743234] Lustre: server umount lustre-OST0000 complete [ 3182.193923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3182.611848] LustreError: 106110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3182.683672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3187.684963] LustreError: 106111:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3187.880925] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3192.806582] LustreError: 106110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3197.030075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3197.356046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3202.416494] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3210.481606] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3216.245313] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3223.250251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3223.692938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3227.841412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3229.776317] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3237.252162] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3248.610839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3248.619569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3248.636608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3252.707507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3252.735513] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3254.849672] Lustre: server umount lustre-OST0000 complete [ 3262.947813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3262.962242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3262.967508] Lustre: Skipped 1 previous similar message [ 3262.975561] Lustre: Skipped 2 previous similar messages [ 3265.385909] Lustre: server umount lustre-MDT0000 complete [ 3268.064232] LustreError: 106110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3268.094983] LustreError: 106110:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3269.200514] LustreError: 106091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118344 with bad export cookie 6333586130906017449 [ 3269.208403] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3269.230016] LustreError: 106091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3269.580227] Lustre: server umount lustre-MDT0001 complete [ 3283.370972] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 3286.782296] Key type lgssc unregistered [ 3287.255667] LNet: 109646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3287.273389] LNetError: 109646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3287.319691] LNet: Removed LNI 192.168.203.144@tcp [ 3289.064930] Key type .llcrypt unregistered [ 3289.071123] Key type ._llcrypt unregistered [ 3319.906698] Key type ._llcrypt registered [ 3319.914883] Key type .llcrypt registered [ 3320.059497] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3321.519903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3321.641342] alg: No test for adler32 (adler32-zlib) [ 3322.986628] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 3323.345912] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 3325.127313] Key type lgssc registered [ 3326.611641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3336.205440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3336.222934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3337.665396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3343.250686] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3346.977572] LustreError: 111067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3347.016259] LustreError: 111067:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3352.046326] LustreError: 111068:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3353.654961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3354.471717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3360.328660] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3368.967835] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3375.176421] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3382.653872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3383.148527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3390.090388] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3397.819176] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3407.735918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3412.612424] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3416.903781] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3423.208112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3423.209550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3423.210237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3423.218734] Lustre: Skipped 1 previous similar message [ 3428.034902] Lustre: server umount lustre-OST0000 complete [ 3433.441593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3433.446865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3433.460367] Lustre: Skipped 1 previous similar message [ 3433.468806] Lustre: Skipped 2 previous similar messages [ 3438.562295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3438.574113] Lustre: Skipped 1 previous similar message [ 3438.778832] Lustre: server umount lustre-MDT0000 complete [ 3443.243239] LustreError: 111049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118518 with bad export cookie 14609891812210654352 [ 3443.262551] LustreError: 111049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3443.270267] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3443.820649] Lustre: server umount lustre-MDT0001 complete [ 3456.572522] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 3459.513994] Key type lgssc unregistered [ 3459.872255] LNet: 114734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3459.889046] LNetError: 114734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3459.917671] LNet: Removed LNI 192.168.203.144@tcp [ 3460.832704] Key type .llcrypt unregistered [ 3460.835513] Key type ._llcrypt unregistered [ 3478.716202] Key type ._llcrypt registered [ 3478.719982] Key type .llcrypt registered [ 3478.834950] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3480.076933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3480.277097] alg: No test for adler32 (adler32-zlib) [ 3481.325780] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 3481.703443] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 3483.471178] Key type lgssc registered [ 3484.929666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3494.393454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3494.413430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3496.000358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3500.730634] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3505.121311] LustreError: 115957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3505.141962] LustreError: 115957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3510.244560] LustreError: 115958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3510.526806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3511.195677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3516.085491] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3524.787169] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3531.675993] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3539.072845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3539.591823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3540.590196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3547.574996] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3558.358109] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3571.704046] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3571.708772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3571.715622] LustreError: Skipped 1 previous similar message [ 3571.719611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3571.745328] Lustre: Skipped 1 previous similar message [ 3571.773748] Lustre: Skipped 1 previous similar message [ 3574.233431] Lustre: server umount lustre-OST0000 complete [ 3581.928984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3581.935198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3581.966219] Lustre: Skipped 1 previous similar message [ 3584.501273] Lustre: server umount lustre-MDT0000 complete [ 3587.041512] LustreError: 115957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3587.064041] LustreError: 115957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3588.479430] LustreError: 115939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118663 with bad export cookie 12126805365499356772 [ 3588.486931] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3588.502301] LustreError: 115939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3589.049752] Lustre: server umount lustre-MDT0001 complete [ 3603.274746] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 3606.489311] Key type lgssc unregistered [ 3606.848778] LNet: 119493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3606.859757] LNetError: 119493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3607.912174] LNet: Removed LNI 192.168.203.144@tcp [ 3609.026881] Key type .llcrypt unregistered [ 3609.031363] Key type ._llcrypt unregistered [ 3634.910392] Key type ._llcrypt registered [ 3634.915302] Key type .llcrypt registered [ 3635.076156] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3636.269723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3636.342309] alg: No test for adler32 (adler32-zlib) [ 3637.455768] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 3637.820624] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 3639.543397] Key type lgssc registered [ 3641.055584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3650.013528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3650.032462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3651.582439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3656.834511] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3661.797532] LustreError: 120911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3661.810508] LustreError: 120911:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3666.453754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3666.931660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3671.531565] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3679.776215] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3686.772702] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3694.482736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3694.845430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3696.897872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3702.297703] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3710.507694] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3718.432962] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3722.491664] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3722.501487] Lustre: Skipped 2 previous similar messages [ 3727.415073] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3727.421375] Lustre: Skipped 2 previous similar messages [ 3727.456889] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3727.500033] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3734.289062] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.44@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3734.296246] Lustre: Skipped 2 previous similar messages [ 3735.946273] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.44@tcp 192.168.0.[2,10]@tcp [ 3740.683306] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.44/24@tcp in log lustre-MDT0000 [ 3740.691163] Lustre: Skipped 2 previous similar messages [ 3744.160193] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.44/24@tcp [ 3744.169277] Lustre: Skipped 1 previous similar message [ 3752.435659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3752.441408] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3752.443521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3752.449158] Lustre: Skipped 1 previous similar message [ 3754.511086] Lustre: server umount lustre-OST0000 complete [ 3762.660320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3762.664672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3762.682567] Lustre: Skipped 2 previous similar messages [ 3762.704031] Lustre: Skipped 1 previous similar message [ 3766.133483] Lustre: server umount lustre-MDT0000 complete [ 3767.794249] LustreError: 120911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3767.827915] LustreError: 120911:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3771.263651] LustreError: 124402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118846 with bad export cookie 13002939574193266950 [ 3771.265169] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3771.310518] LustreError: 124402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3771.910249] Lustre: server umount lustre-MDT0001 complete [ 3784.262807] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 3787.361401] Key type lgssc unregistered [ 3787.696304] LNet: 125609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3787.714206] LNetError: 125609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3787.771168] LNet: Removed LNI 192.168.203.144@tcp [ 3788.691427] Key type .llcrypt unregistered [ 3788.693467] Key type ._llcrypt unregistered [ 3820.820191] Key type ._llcrypt registered [ 3820.822467] Key type .llcrypt registered [ 3820.911527] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3822.049376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3822.176679] alg: No test for adler32 (adler32-zlib) [ 3823.327444] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 3823.614736] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 3825.367742] Key type lgssc registered [ 3826.883617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3834.987894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3835.003673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3836.370234] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3836.400450] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.44/24@tcp [ 3836.432531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3841.584083] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3846.635055] LustreError: 127168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3846.654776] LustreError: 127168:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3850.853140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3851.225215] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3851.264473] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.44/24@tcp [ 3851.303530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3856.350680] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3865.329332] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3871.861610] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3878.936527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3879.438181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3881.540858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3886.239735] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3894.794759] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3907.045862] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3907.051790] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3907.074520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3907.553704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3907.598092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3912.477402] Lustre: server umount lustre-OST0000 complete [ 3917.796119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3917.796212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3917.817583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3917.841891] Lustre: Skipped 1 previous similar message [ 3922.925723] LustreError: 127168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3922.952053] LustreError: 127168:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3923.081821] Lustre: server umount lustre-MDT0000 complete [ 3927.008502] LustreError: 127165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3927.019904] LustreError: 127165:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3927.316743] LustreError: 127150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119002 with bad export cookie 11185036997387213114 [ 3927.333383] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3927.334266] LustreError: 127150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3927.607673] Lustre: server umount lustre-MDT0001 complete [ 3941.808659] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 3945.154982] Key type lgssc unregistered [ 3945.517812] LNet: 130758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3945.525731] LNetError: 130758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3945.545738] LNet: Removed LNI 192.168.203.144@tcp [ 3946.499444] Key type .llcrypt unregistered [ 3946.513035] Key type ._llcrypt unregistered [ 3976.376386] Key type ._llcrypt registered [ 3976.378980] Key type .llcrypt registered [ 3976.507084] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 3997.091609] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3998.653714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3998.692997] alg: No test for adler32 (adler32-zlib) [ 4000.156741] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 4000.444208] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 4002.199977] Key type lgssc registered [ 4003.639272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4041.980387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4042.035943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4043.428230] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4043.493452] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4043.650445] Lustre: lustre-MDT0000: new disk, initializing [ 4043.757363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4043.779892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4049.080483] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4060.249470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4060.337274] Lustre: 134161:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4060.413961] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4060.417360] Lustre: Skipped 1 previous similar message [ 4060.546285] Lustre: lustre-MDT0001: new disk, initializing [ 4060.651606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4060.696901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4060.719029] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4065.441915] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4075.296244] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4081.853518] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4089.734868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4090.118763] Lustre: lustre-OST0000: new disk, initializing [ 4090.121922] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4090.127112] Lustre: 136134:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4090.212472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4097.061891] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4099.185078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4099.191240] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4099.328113] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4107.173567] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4112.242913] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4112.447821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4117.391817] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4117.640374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4120.037760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4120.044306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4120.058907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4120.080177] Lustre: Skipped 1 previous similar message [ 4125.153214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4125.165471] Lustre: Skipped 1 previous similar message [ 4130.290585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4134.367215] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4134.557089] Lustre: server umount lustre-OST0000 complete [ 4140.513286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4140.526360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4140.551691] Lustre: Skipped 2 previous similar messages [ 4140.571061] Lustre: Skipped 2 previous similar messages [ 4145.634996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4145.648877] Lustre: Skipped 1 previous similar message [ 4153.311166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4153.530161] Lustre: server umount lustre-MDT0000 complete [ 4158.041117] LustreError: 134156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119233 with bad export cookie 4325205352174556086 [ 4158.049533] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4158.052230] LustreError: 134156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4158.683699] Lustre: server umount lustre-MDT0001 complete [ 4167.589419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4167.975238] LustreError: 138353:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4167.996360] LustreError: 138353:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4168.052419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4173.153867] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4173.285075] LustreError: 138354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4173.304910] LustreError: 138354:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4178.403086] LustreError: 138353:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4183.261563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4183.531900] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4183.544714] Lustre: Skipped 2 previous similar messages [ 4183.849923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4189.388177] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4198.204439] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4204.782896] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4211.937180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4212.283671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4219.180185] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4227.642101] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4232.561770] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4232.767872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4237.500374] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4237.724671] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4246.957803] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4247.177227] Lustre: lustre-OST0001: new disk, initializing [ 4247.182667] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4247.189424] Lustre: 141733:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4247.283406] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4254.396783] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4254.767146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4254.774632] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4254.849946] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4263.317363] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4263.563449] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4269.813652] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4270.262406] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4279.071516] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff905106062800.ost_server_uuid 50 [ 4281.442728] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff905106062800.ost_server_uuid in IDLE state after 0 sec [ 4295.654966] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4295.657373] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4295.664230] LustreError: Skipped 1 previous similar message [ 4295.678315] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4295.693378] Lustre: Skipped 1 previous similar message [ 4298.164917] Lustre: server umount lustre-OST0001 complete [ 4300.780707] LustreError: 142116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4300.801904] LustreError: 142116:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4305.383389] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4305.385517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4305.390407] LustreError: Skipped 1 previous similar message [ 4305.419469] Lustre: Skipped 2 previous similar messages [ 4305.892335] LustreError: 140187:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4305.911971] LustreError: 140187:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4308.908478] Lustre: server umount lustre-OST0000 complete [ 4316.130596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4316.150695] Lustre: Skipped 1 previous similar message [ 4327.392028] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4327.575773] Lustre: server umount lustre-MDT0000 complete [ 4331.489397] LustreError: 139319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4331.515901] LustreError: 139319:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4331.838491] LustreError: 138334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119406 with bad export cookie 4325205352174556968 [ 4331.842481] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4331.847882] LustreError: 138334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4332.190829] Lustre: server umount lustre-MDT0001 complete [ 4352.657748] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 4355.738780] Key type lgssc unregistered [ 4356.047301] LNet: 144740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4356.056694] LNetError: 144740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4356.083218] LNet: Removed LNI 192.168.203.144@tcp [ 4357.075503] Key type .llcrypt unregistered [ 4357.079636] Key type ._llcrypt unregistered [ 4399.861573] Key type ._llcrypt registered [ 4399.864582] Key type .llcrypt registered [ 4399.994139] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 4416.097426] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 4417.522021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4417.576202] alg: No test for adler32 (adler32-zlib) [ 4418.751877] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 4418.994986] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 4420.672686] Key type lgssc registered [ 4421.905319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4462.774977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4462.803369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4464.170968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4464.210918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4464.374371] Lustre: lustre-MDT0000: new disk, initializing [ 4464.527597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4464.559915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4469.615365] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4481.148847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4481.239582] Lustre: 148901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4481.292473] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4481.296157] Lustre: Skipped 1 previous similar message [ 4481.362168] Lustre: lustre-MDT0001: new disk, initializing [ 4481.511122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4481.579300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4481.599899] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4488.558690] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4500.062329] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4507.310201] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4514.407563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4514.701206] Lustre: lustre-OST0000: new disk, initializing [ 4514.704605] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4514.710126] Lustre: 150868:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4514.805123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4521.838188] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4524.116094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4524.122926] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4524.305120] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4531.958505] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4537.273732] Lustre: Failing over lustre-OST0000 [ 4537.491731] Lustre: server umount lustre-OST0000 complete [ 4554.146151] Lustre: 147126:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787119613/real 1787119613] req@0000000065b030b9 x1873930610574848/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787119629 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4554.200682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4555.234256] Lustre: 147126:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787119614/real 1787119614] req@0000000045e608f2 x1873930610574976/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787119630 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4555.256158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4556.319136] Lustre: 147124:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787119614/real 1787119614] req@00000000d149b996 x1873930610575104/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787119630 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4556.931800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4557.383354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4557.412555] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4561.439125] Lustre: 147125:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787119620/real 1787119620] req@0000000099bdf8f1 x1873930610576896/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787119636 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4561.480414] Lustre: 147125:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4563.859949] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4564.529693] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4565.679139] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4565.682090] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4566.559128] Lustre: 147125:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787119625/real 1787119625] req@00000000d8468a11 x1873930610578048/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787119641 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4568.150762] Lustre: Failing over lustre-MDT0000 [ 4568.548630] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4568.556023] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4568.707039] Lustre: server umount lustre-MDT0000 complete [ 4574.692615] LustreError: 148910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4574.712633] LustreError: 148910:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4575.819619] LustreError: 148907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4579.810973] LustreError: 148905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4579.841627] LustreError: 148905:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4584.936084] LustreError: 148906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4584.975755] LustreError: 148906:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4585.429449] Lustre: 147124:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787119644/real 1787119644] req@000000002318cb2c x1873930610590720/t0(0) o400->MGC192.168.203.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787119660 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4585.474755] Lustre: 147124:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4585.488094] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4588.417506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4590.563041] LustreError: 148911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4590.595175] LustreError: 148911:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4596.045177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4596.138867] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4596.309515] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4601.347660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4601.359567] Lustre: Skipped 1 previous similar message [ 4601.382303] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4601.431129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4601.645591] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4611.564259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4611.567275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4611.582385] Lustre: Skipped 3 previous similar messages [ 4616.678307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4616.687976] Lustre: Skipped 2 previous similar messages [ 4617.403728] Lustre: server umount lustre-OST0000 complete [ 4621.795766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4621.801306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4621.801682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4621.811845] Lustre: Skipped 1 previous similar message [ 4626.915970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4626.927019] Lustre: Skipped 2 previous similar messages [ 4627.527750] Lustre: server umount lustre-MDT0000 complete [ 4631.211668] LustreError: 148891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119706 with bad export cookie 5830216144203332851 [ 4631.219947] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4631.230272] LustreError: 148891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4631.610268] Lustre: server umount lustre-MDT0001 complete [ 4642.283780] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 4644.675699] Key type lgssc unregistered [ 4644.900950] LNet: 154523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4644.920279] LNetError: 154523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4645.989715] LNet: Removed LNI 192.168.203.144@tcp [ 4646.812847] Key type .llcrypt unregistered [ 4646.814809] Key type ._llcrypt unregistered [ 4682.478828] Key type ._llcrypt registered [ 4682.481756] Key type .llcrypt registered [ 4682.583292] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 4683.594276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4683.781839] alg: No test for adler32 (adler32-zlib) [ 4684.822650] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 4685.055591] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 4686.832166] Key type lgssc registered [ 4688.141863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4696.356097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4696.381121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4697.976564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4703.372733] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4708.325223] LustreError: 156044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4708.361118] LustreError: 156044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4713.447036] LustreError: 156045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4713.630586] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4717.801036] LustreError: 156073:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4717.806204] LustreError: 156073:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4717.811213] LustreError: 156073:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4718.113894] Lustre: server umount lustre-MDT0000 complete [ 4732.386783] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 4740.455691] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 4775.373484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4775.646318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4775.704183] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4775.835879] Lustre: lustre-MDT0000: new disk, initializing [ 4775.948397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4775.969363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4780.818854] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4791.297270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4791.445431] Lustre: 159375:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4791.547831] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4791.557494] Lustre: Skipped 1 previous similar message [ 4791.645495] Lustre: lustre-MDT0001: new disk, initializing [ 4791.755066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4791.790047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4791.802556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4796.459440] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4806.778332] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4813.624645] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4821.291560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4821.521928] Lustre: lustre-OST0000: new disk, initializing [ 4821.525649] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4821.531571] Lustre: 161345:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4821.615293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4828.779797] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4831.344318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4831.350196] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4831.431293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4838.401657] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4842.602201] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4845.041281] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:11:58 (1787119918) [ 5118.431598] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5118.435561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5118.446247] Lustre: Skipped 1 previous similar message [ 5122.846899] Lustre: server umount lustre-MDT0000 complete [ 5123.554155] LustreError: 159384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5123.573863] LustreError: 159384:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5126.525952] LustreError: 159370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120201 with bad export cookie 10749079040887962364 [ 5126.526195] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5126.547212] LustreError: 159370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5128.672725] LustreError: 159382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5128.672923] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5128.718978] Lustre: Skipped 1 previous similar message [ 5128.723161] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5128.731246] Lustre: Skipped 2 previous similar messages [ 5131.679801] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5132.932454] Lustre: server umount lustre-MDT0001 complete [ 5142.836600] Lustre: server umount lustre-OST0000 complete [ 5151.892472] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 5161.482801] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5196.599342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5196.832785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5196.855255] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5196.920340] Lustre: lustre-MDT0000: new disk, initializing [ 5196.976761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5196.989175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5200.642124] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5209.572747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5209.706205] Lustre: 166009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5209.743831] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5209.748140] Lustre: Skipped 1 previous similar message [ 5209.847771] Lustre: lustre-MDT0001: new disk, initializing [ 5209.949928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5210.006428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5210.018126] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5214.184025] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5223.956661] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5230.200451] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5237.430663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5237.708169] Lustre: lustre-OST0000: new disk, initializing [ 5237.715448] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5237.724323] Lustre: 167976:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5237.841410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5243.980042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5243.991522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5244.197195] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5245.111544] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5254.747510] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5259.177161] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5259.441750] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5264.584872] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5264.973981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5269.984044] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5269.992726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5269.996698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5270.030630] Lustre: Skipped 1 previous similar message [ 5275.144813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5275.153080] Lustre: Skipped 2 previous similar messages [ 5281.247435] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5281.651158] Lustre: server umount lustre-OST0000 complete [ 5286.881380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5286.890607] LustreError: Skipped 1 previous similar message [ 5286.894416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5286.920172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5286.923052] Lustre: Skipped 2 previous similar messages [ 5292.800866] Lustre: server umount lustre-MDT0000 complete [ 5295.593656] LustreError: 166898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5295.646672] LustreError: 166898:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5296.656369] LustreError: 166003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120371 with bad export cookie 10749079040888121194 [ 5296.671108] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5296.673381] LustreError: 166003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5297.230440] Lustre: server umount lustre-MDT0001 complete [ 5306.580950] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:19:40 (1787120380) [ 5322.916835] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 5333.132867] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5371.157443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5371.404472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5371.433770] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5371.579877] Lustre: lustre-MDT0000: new disk, initializing [ 5371.666536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5371.683881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5376.498618] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5386.827070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5386.957972] Lustre: 172732:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5387.175115] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5391.847807] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5401.937616] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5408.330690] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5415.388901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5415.717846] Lustre: lustre-OST0000: new disk, initializing [ 5415.720666] Lustre: Skipped 1 previous similar message [ 5415.730266] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5415.739559] Lustre: Skipped 2 previous similar messages [ 5415.744745] Lustre: 174708:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5422.140242] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5425.771856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5425.788314] Lustre: Skipped 1 previous similar message [ 5425.795227] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5425.871970] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5431.240888] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5441.508128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5441.510072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5441.515462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5441.515469] Lustre: Skipped 2 previous similar messages [ 5441.539612] LustreError: Skipped 1 previous similar message [ 5441.540689] Lustre: Skipped 2 previous similar messages [ 5443.745163] Lustre: server umount lustre-OST0000 complete [ 5448.673948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5451.745389] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5451.761646] Lustre: Skipped 1 previous similar message [ 5453.536832] Lustre: server umount lustre-MDT0000 complete [ 5456.724240] LustreError: 172724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120531 with bad export cookie 10749079040888122076 [ 5456.735207] LustreError: 172724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5456.735708] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5457.098532] Lustre: server umount lustre-MDT0001 complete [ 5463.886959] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:22:18 (1787120538) [ 5478.574421] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 5487.497636] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5522.753906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5522.898616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5522.923055] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5523.009768] Lustre: lustre-MDT0000: new disk, initializing [ 5523.207898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5523.217532] Lustre: Skipped 2 previous similar messages [ 5523.248199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5527.430746] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5539.258753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5539.560766] Lustre: 179034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5540.018138] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5544.272970] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5555.240626] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5560.932327] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5568.713566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5569.078762] Lustre: 181010:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5577.178730] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5578.371881] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5578.588872] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5589.203644] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5599.207289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5599.216084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5599.220433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5599.230762] Lustre: Skipped 1 previous similar message [ 5599.243774] Lustre: Skipped 5 previous similar messages [ 5602.190983] Lustre: server umount lustre-OST0000 complete [ 5620.191455] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5620.464504] Lustre: server umount lustre-MDT0000 complete [ 5623.858422] LustreError: 179026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120698 with bad export cookie 10749079040888123161 [ 5623.864123] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5623.874276] LustreError: 179026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5624.176657] Lustre: server umount lustre-MDT0001 complete [ 5638.103892] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 5641.148097] Key type lgssc unregistered [ 5641.537909] LNet: 182869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5641.558350] LNetError: 182869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5642.598895] LNet: Removed LNI 192.168.203.144@tcp [ 5643.901327] Key type .llcrypt unregistered [ 5643.902961] Key type ._llcrypt unregistered [ 5667.387331] Key type ._llcrypt registered [ 5667.389951] Key type .llcrypt registered [ 5667.525470] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5668.699350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5668.733318] alg: No test for adler32 (adler32-zlib) [ 5669.851825] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 5670.194287] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 5671.943172] Key type lgssc registered [ 5673.346720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5680.602952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5680.618356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5682.117351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5686.468960] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5692.387060] LustreError: 184285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5692.416684] LustreError: 184285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5696.672895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5697.486359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5703.378137] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5712.314570] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5718.425592] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5726.034532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5726.442613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5727.490939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5733.130107] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5740.944344] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5763.552220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5763.562497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5763.564692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5763.592401] Lustre: Skipped 1 previous similar message [ 5768.189461] Lustre: server umount lustre-OST0000 complete [ 5773.809128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5773.817302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5773.835348] Lustre: Skipped 1 previous similar message [ 5773.860687] Lustre: Skipped 2 previous similar messages [ 5778.316876] Lustre: server umount lustre-MDT0000 complete [ 5778.915733] LustreError: 184285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5778.929877] LustreError: 184285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5782.447404] LustreError: 184266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120857 with bad export cookie 14954060353116163666 [ 5782.453147] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5782.458081] LustreError: 184266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5782.833892] Lustre: server umount lustre-MDT0001 complete [ 5794.415962] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 5796.996802] Key type lgssc unregistered [ 5797.268765] LNet: 187841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5797.284800] LNetError: 187841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5797.305798] LNet: Removed LNI 192.168.203.144@tcp [ 5798.088210] Key type .llcrypt unregistered [ 5798.090770] Key type ._llcrypt unregistered [ 5824.313323] Key type ._llcrypt registered [ 5824.318115] Key type .llcrypt registered [ 5824.492185] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5825.698114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5825.820112] alg: No test for adler32 (adler32-zlib) [ 5827.060197] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 5827.532952] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 5829.239617] Key type lgssc registered [ 5831.045056] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5840.235958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5840.256200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5841.650407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5845.622251] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5851.111573] LustreError: 189260:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5851.121886] LustreError: 189260:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5853.473358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5853.843645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5858.322407] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5865.702570] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5871.204371] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5877.311712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5877.670861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5883.384073] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5886.908969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5891.587754] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5899.241305] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5899.248597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5899.260880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5902.304026] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5902.311808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5902.322246] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5902.334702] Lustre: Skipped 1 previous similar message [ 5904.457592] Lustre: lustre-OST0000: Not available for connect from 192.168.203.44@tcp (stopping) [ 5904.468461] Lustre: Skipped 1 previous similar message [ 5904.691469] Lustre: server umount lustre-OST0000 complete [ 5912.382258] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9051075ec800.ost_server_uuid 50 [ 5914.396122] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5930.473924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5930.483032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5930.500823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5933.031320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5934.983879] Lustre: server umount lustre-MDT0000 complete [ 5938.145529] LustreError: 189255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5938.177901] LustreError: 189255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5938.479799] LustreError: 189240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121013 with bad export cookie 514750023060696992 [ 5938.489937] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5938.493571] LustreError: 189240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5938.789399] Lustre: server umount lustre-MDT0001 complete [ 5947.939458] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:30:21 (1787121021) [ 5954.708430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5955.208667] LustreError: 193175:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5955.306340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5960.427480] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5960.678488] LustreError: 193176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5965.792071] LustreError: 193175:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5970.169624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5970.660607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5976.083797] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5984.602946] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5991.131487] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5999.713434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6000.423596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6001.478261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6009.554213] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6018.631517] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6026.868244] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6027.094866] Lustre: lustre-OST0001: new disk, initializing [ 6027.100694] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6027.108416] Lustre: 196018:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6027.192675] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6032.470373] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6033.586283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6033.610798] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6033.789421] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6042.181158] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6046.688703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6046.695430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6046.716147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6046.727095] Lustre: Skipped 2 previous similar messages [ 6051.009381] Lustre: server umount lustre-OST0000 complete [ 6053.454716] LustreError: 194998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6057.358381] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6057.580926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6062.831819] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6063.102476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6069.739534] LustreError: 196427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6069.776178] LustreError: 196427:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6079.973734] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6079.980087] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6079.980674] LustreError: Skipped 2 previous similar messages [ 6079.985520] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6079.992762] Lustre: Skipped 2 previous similar messages [ 6080.009532] Lustre: Skipped 5 previous similar messages [ 6083.404060] Lustre: server umount lustre-OST0001 complete [ 6088.672150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6094.089676] Lustre: server umount lustre-MDT0000 complete [ 6097.897842] LustreError: 193157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121172 with bad export cookie 514750023060698063 [ 6097.916509] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6097.928383] LustreError: 193157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6098.360455] Lustre: server umount lustre-MDT0001 complete [ 6124.878427] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:33:18 (1787121198) [ 6132.210740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6132.469698] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6132.499966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6132.675554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6137.641402] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6146.355600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6146.485536] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6146.515842] Lustre: 199457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6151.235675] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6159.569642] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6165.995837] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6172.629731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6172.876153] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6178.422146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6179.142959] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6187.443905] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6194.146462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6194.251484] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6201.167774] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6208.819100] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6214.113229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6214.131381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6214.142185] Lustre: Skipped 2 previous similar messages [ 6214.148237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6214.153903] Lustre: Skipped 3 previous similar messages [ 6226.403708] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6226.548460] Lustre: server umount lustre-OST0000 complete [ 6230.095945] LustreError: 201307:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6230.121448] LustreError: 201307:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6246.881774] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6246.905163] LustreError: Skipped 1 previous similar message [ 6246.906055] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6246.916433] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6246.931220] Lustre: Skipped 2 previous similar messages [ 6246.956836] Lustre: Skipped 11 previous similar messages [ 6249.199722] Lustre: server umount lustre-OST0001 complete [ 6259.700070] Lustre: server umount lustre-MDT0000 complete [ 6263.500052] LustreError: 199449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121338 with bad export cookie 514750023060699470 [ 6263.510908] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6263.518819] LustreError: 199449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6263.969332] Lustre: server umount lustre-MDT0001 complete [ 6285.259652] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:35:59 (1787121359) [ 6301.485920] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 6311.112435] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 6349.598097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6350.092880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6350.142745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6350.313386] Lustre: lustre-MDT0000: new disk, initializing [ 6350.442832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6350.450471] Lustre: Skipped 3 previous similar messages [ 6350.484408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6357.097131] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6369.503472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6369.584803] Lustre: 207554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6369.651622] LustreError: 207566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6369.673344] LustreError: 207566:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 6369.707747] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6369.715020] Lustre: Skipped 1 previous similar message [ 6369.790478] Lustre: lustre-MDT0001: new disk, initializing [ 6369.942369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6369.954282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6374.459717] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6385.048924] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6390.112849] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6396.355464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6396.661930] Lustre: lustre-OST0000: new disk, initializing [ 6396.666938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6396.678947] Lustre: 209525:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6403.140233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6403.154960] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6403.318870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6403.472584] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6413.048503] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6417.256552] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6417.464836] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6421.965178] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6422.202380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6429.167363] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6429.168234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6429.176577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6429.176585] Lustre: Skipped 3 previous similar messages [ 6429.178108] LustreError: Skipped 2 previous similar messages [ 6429.223939] Lustre: Skipped 3 previous similar messages [ 6438.367977] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6438.663263] Lustre: server umount lustre-OST0000 complete [ 6448.882276] Lustre: server umount lustre-MDT0000 complete [ 6452.547214] LustreError: 208351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121527 with bad export cookie 514750023060701038 [ 6452.548461] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6452.562118] LustreError: 208351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6452.993749] Lustre: server umount lustre-MDT0001 complete [ 6462.181513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6468.184367] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6477.456114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6482.285537] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6490.430862] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6496.227913] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6502.882588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6503.258772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6503.269524] Lustre: Skipped 4 previous similar messages [ 6510.786442] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6519.220259] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6524.429390] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6524.655525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6529.542578] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6529.729509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6537.890650] Lustre: server umount lustre-OST0000 complete [ 6544.805274] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6549.637334] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6578.066039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6584.173781] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6591.592739] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6597.453939] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 6598.950599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6598.968234] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6598.974569] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6598.988125] Lustre: Skipped 1 previous similar message [ 6604.258993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6604.265831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6604.269466] LustreError: Skipped 3 previous similar messages [ 6604.271507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6604.271513] Lustre: Skipped 7 previous similar messages [ 6604.295103] Lustre: Skipped 5 previous similar messages [ 6607.612988] Lustre: server umount lustre-OST0000 complete [ 6617.491287] Lustre: server umount lustre-MDT0000 complete [ 6619.621901] LustreError: 213758:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6619.650829] LustreError: 213758:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6621.063300] LustreError: 211725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121696 with bad export cookie 514750023060701920 [ 6621.065853] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6621.074054] LustreError: 211725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6621.392883] Lustre: server umount lustre-MDT0001 complete [ 6629.196486] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:41:43 (1787121703) [ 6634.835360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6639.457767] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6647.804763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6652.909503] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6662.172264] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6668.985756] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6676.356598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6683.482323] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6690.648477] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6695.619857] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6695.916944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6700.994118] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6701.226340] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6707.952451] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6708.091548] Lustre: lustre-OST0001: new disk, initializing [ 6708.095396] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6708.103966] Lustre: 221080:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6713.001155] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6714.427842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6714.447568] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6714.571769] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6721.587694] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6725.721941] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6725.903257] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6729.817451] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6730.132381] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6746.079110] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6746.210088] Lustre: server umount lustre-OST0001 complete [ 6752.536634] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6752.838985] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6757.682989] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6757.915972] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6786.512493] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6786.726485] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6786.730138] Lustre: Skipped 5 previous similar messages [ 6792.177912] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6792.208365] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6792.211156] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6792.273768] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6799.790948] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6801.403198] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6817.247397] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6817.467575] Lustre: server umount lustre-OST0001 complete [ 6848.991239] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6849.122164] Lustre: server umount lustre-OST0000 complete [ 6858.801375] Lustre: server umount lustre-MDT0000 complete [ 6861.822837] LustreError: 221079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121936 with bad export cookie 514750023060702879 [ 6861.824017] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6861.840025] LustreError: 221079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6868.173557] Lustre: server umount lustre-MDT0001 complete [ 6891.031732] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 02:46:05 (1787121965) [ 6897.258712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6897.558201] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6897.581500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6902.318520] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6909.967285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6910.045340] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6910.065567] Lustre: 226333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6914.453280] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6921.219613] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6926.711466] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6933.292487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6933.533602] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6938.946588] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6945.136474] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6953.205928] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6953.257598] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6958.499670] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6965.380846] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6966.627300] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6970.723595] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6970.958584] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6975.196554] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6975.412313] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6980.888639] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9051075ee000.ost_server_uuid 50 [ 6982.343991] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9051075ee000.ost_server_uuid in IDLE state after 0 sec [ 6983.215166] Lustre: Permanently deactivating lustre-OST0001 [ 6983.241582] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6986.594852] Lustre: Permanently reactivating lustre-OST0001 [ 6994.925901] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6994.931921] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6994.940948] Lustre: Skipped 11 previous similar messages [ 6994.952316] Lustre: Skipped 25 previous similar messages [ 6994.961372] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6994.971188] LustreError: Skipped 5 previous similar messages [ 6996.968408] Lustre: server umount lustre-OST0001 complete [ 7000.044866] LustreError: 228664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7000.068910] LustreError: 228664:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 43 previous similar messages [ 7006.796930] Lustre: server umount lustre-OST0000 complete [ 7016.925587] Lustre: server umount lustre-MDT0000 complete [ 7020.558467] LustreError: 228190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122095 with bad export cookie 514750023060704174 [ 7020.567802] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7020.576524] LustreError: 228190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7021.033569] Lustre: server umount lustre-MDT0001 complete [ 7043.615962] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 02:48:38 (1787122118) [ 7050.528413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7050.765906] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7050.792826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7050.799486] Lustre: Skipped 5 previous similar messages [ 7054.728731] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7062.035790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7062.133132] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7062.160778] Lustre: 233026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7065.589906] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7072.060588] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7077.235273] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7082.738764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7082.946247] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7082.965454] Lustre: Permanently deactivating lustre-OST0000 [ 7082.977208] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7088.473785] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7093.270705] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7094.798907] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7101.181573] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7101.260374] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7101.499148] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7106.991261] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7113.412138] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7115.170875] Lustre: Permanently reactivating lustre-OST0000 [ 7115.195743] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7115.204275] Lustre: Skipped 2 previous similar messages [ 7120.953609] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7120.964142] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7120.990427] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7120.999606] Lustre: Skipped 1 previous similar message [ 7121.009463] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7121.029297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7131.002569] Lustre: server umount lustre-OST0001 complete [ 7140.632883] Lustre: server umount lustre-OST0000 complete [ 7150.374560] Lustre: server umount lustre-MDT0000 complete [ 7153.482689] LustreError: 233016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122228 with bad export cookie 514750023060705847 [ 7153.483841] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7153.498846] LustreError: 233016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7153.739633] Lustre: server umount lustre-MDT0001 complete [ 7163.754864] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7165.985188] Key type lgssc unregistered [ 7166.238745] LNet: 237905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7166.247752] LNetError: 237905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7166.267228] LNet: Removed LNI 192.168.203.144@tcp [ 7167.038443] Key type .llcrypt unregistered [ 7167.041414] Key type ._llcrypt unregistered [ 7186.927141] Key type ._llcrypt registered [ 7186.929784] Key type .llcrypt registered [ 7187.010309] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7187.859635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7187.899195] alg: No test for adler32 (adler32-zlib) [ 7188.917394] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 7189.142109] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 7190.823192] Key type lgssc registered [ 7191.831470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7200.817664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7200.839164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7202.297568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7207.354336] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7212.518384] LustreError: 239351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7212.532308] LustreError: 239351:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7215.762767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7215.913060] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7215.961890] Lustre: 239341:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7215.971615] Lustre: Permanently deactivating lustre-MDT0001 [ 7215.996450] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7216.382660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7218.497483] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7218.508306] LustreError: 239382:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7218.518907] LustreError: 239382:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7218.526858] LustreError: 239382:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 7221.250732] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7229.020750] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7234.250505] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7241.834224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7242.491120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7247.589748] LustreError: 241192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7247.599041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7248.984821] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7252.964484] LustreError: 241192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7257.567362] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7258.086227] LustreError: 241195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7261.176196] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 7264.514248] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7264.772221] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7269.880261] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7269.888498] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7271.781131] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7279.542720] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7283.387391] Lustre: Permanently reactivating lustre-MDT0001 [ 7283.423876] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7283.428561] Lustre: Skipped 2 previous similar messages [ 7286.712898] Lustre: lustre-MDT0001: Client be7b2feb-6c7d-4242-a926-a8885eb395a4 (at 192.168.203.44@tcp) reconnecting [ 7292.595303] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7294.294367] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7300.540547] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7300.810371] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7304.969668] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7305.144242] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7308.110410] Lustre: Permanently deactivating lustre-MDT0001 [ 7308.153621] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7308.170782] Lustre: Skipped 2 previous similar messages [ 7318.591306] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7329.248294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7329.263085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7329.278882] LustreError: 243845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7329.305280] LustreError: 243845:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7329.434089] Lustre: server umount lustre-MDT0000 complete [ 7331.302105] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7331.315369] Lustre: Skipped 1 previous similar message [ 7333.037894] LustreError: 239333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122408 with bad export cookie 13369501005117615787 [ 7333.042394] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7333.047382] LustreError: 239333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7333.414987] Lustre: server umount lustre-MDT0001 complete [ 7342.881946] Lustre: server umount lustre-OST0000 complete [ 7352.517165] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 02:53:46 (1787122426) [ 7361.494928] Lustre: server umount lustre-OST0001 complete [ 7367.612274] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 7379.640767] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7417.739936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7417.919923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7417.925448] Lustre: Skipped 2 previous similar messages [ 7417.939375] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7418.008744] Lustre: lustre-MDT0000: new disk, initializing [ 7418.072283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7418.092399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7422.306362] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7432.018480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7432.118620] Lustre: 247661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7432.172260] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7432.178205] Lustre: Skipped 1 previous similar message [ 7432.235741] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7432.263291] Lustre: lustre-MDT0001: new disk, initializing [ 7432.361612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7432.408450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7432.420215] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7437.083192] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7448.144474] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7455.607191] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7462.379493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7462.675546] Lustre: lustre-OST0000: new disk, initializing [ 7462.684634] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7462.691848] Lustre: 249626:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7462.786955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7468.602649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7468.615767] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7468.782664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7469.162582] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7478.968901] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7481.351375] LustreError: 247667:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7490.700987] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7490.861417] Lustre: lustre-OST0001: new disk, initializing [ 7490.864594] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7490.874218] Lustre: 250743:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7491.377102] LustreError: 247667:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7491.406398] LustreError: 249058:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7497.288809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7497.305835] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7497.452206] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7498.439291] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7501.423156] LustreError: 249058:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7508.272598] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7509.630915] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7513.065497] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7513.065642] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7513.079366] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7513.092833] Lustre: Skipped 2 previous similar messages [ 7513.102929] Lustre: Skipped 1 previous similar message [ 7517.901434] Lustre: server umount lustre-OST0001 complete [ 7518.180444] LustreError: 249948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7518.199853] LustreError: 249948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7535.584698] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7535.584932] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7535.598253] LustreError: Skipped 2 previous similar messages [ 7535.613544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7535.625876] Lustre: Skipped 1 previous similar message [ 7538.670295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7538.672169] LustreError: 249611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7538.675815] Lustre: Skipped 1 previous similar message [ 7538.715441] LustreError: 249611:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 7541.176818] Lustre: server umount lustre-OST0000 complete [ 7545.312373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7545.327149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7545.343448] Lustre: Skipped 1 previous similar message [ 7545.350612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7553.172434] Lustre: server umount lustre-MDT0000 complete [ 7556.814046] LustreError: 247652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122631 with bad export cookie 13369501005117618825 [ 7556.816845] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7556.826895] LustreError: 247652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7563.192530] Lustre: server umount lustre-MDT0001 complete [ 7573.827218] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7576.360485] Key type lgssc unregistered [ 7576.634443] LNet: 252775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7576.643782] LNetError: 252775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7576.663832] LNet: Removed LNI 192.168.203.144@tcp [ 7577.360226] Key type .llcrypt unregistered [ 7577.362606] Key type ._llcrypt unregistered [ 7619.667570] Key type ._llcrypt registered [ 7619.669911] Key type .llcrypt registered [ 7619.762689] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7621.115771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7621.153101] alg: No test for adler32 (adler32-zlib) [ 7622.281505] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 7622.564619] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 7624.255493] Key type lgssc registered [ 7625.473965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7633.185784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7633.208480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7634.597946] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7634.635889] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7634.827068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7639.125476] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7647.592845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7647.697945] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7647.722800] Lustre: 254986:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7647.955523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7652.046775] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7658.600287] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7664.169588] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7670.301385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7670.560696] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7670.757791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7676.737860] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7680.135688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7684.420498] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7741.408432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7741.422751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7741.449289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7741.931560] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7741.932193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7742.006176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7745.641838] Lustre: server umount lustre-OST0000 complete [ 7750.540282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7768.193550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7768.613863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7775.038290] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7782.163922] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7782.980614] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7783.012788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7783.018058] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7783.034739] Lustre: Skipped 1 previous similar message [ 7823.841967] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7823.847930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7823.856839] Lustre: Skipped 1 previous similar message [ 7823.865305] Lustre: Skipped 1 previous similar message [ 7827.993074] Lustre: server umount lustre-OST0000 complete [ 7832.551721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7832.575704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7832.603343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7837.905572] Lustre: server umount lustre-MDT0000 complete [ 7842.309807] LustreError: 254978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122917 with bad export cookie 16645161582716107318 [ 7842.319544] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7842.319710] LustreError: 254978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7842.581951] Lustre: server umount lustre-MDT0001 complete [ 7856.182469] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7859.562961] Key type lgssc unregistered [ 7859.870971] LNet: 260425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7859.889480] LNetError: 260425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7859.914345] LNet: Removed LNI 192.168.203.144@tcp [ 7860.804324] Key type .llcrypt unregistered [ 7860.805962] Key type ._llcrypt unregistered [ 7889.388634] Key type ._llcrypt registered [ 7889.394494] Key type .llcrypt registered [ 7889.524492] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7890.626802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7890.646872] alg: No test for adler32 (adler32-zlib) [ 7891.787118] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 7892.312682] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 7894.031119] Key type lgssc registered [ 7895.842728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7903.543689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7903.560049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7904.963209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7909.070864] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7914.484606] LustreError: 261842:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7914.506072] LustreError: 261842:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7918.061430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7918.580497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7923.917351] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7931.015443] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7936.660496] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7942.740365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7943.041112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7948.277385] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7949.274144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7955.453934] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7980.011552] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7980.011612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7980.016924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7980.047371] Lustre: Skipped 1 previous similar message [ 7982.958223] Lustre: server umount lustre-OST0000 complete [ 7990.241320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7990.245065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7990.259416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7990.274906] Lustre: Skipped 1 previous similar message [ 7990.284354] Lustre: Skipped 2 previous similar messages [ 7992.738386] Lustre: server umount lustre-MDT0000 complete [ 7995.360502] LustreError: 261837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7995.376395] LustreError: 261837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7995.996303] LustreError: 261824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123071 with bad export cookie 16656859718419972067 [ 7996.002336] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7996.004774] LustreError: 261824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7996.024829] LustreError: 261153:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 7996.283976] Lustre: server umount lustre-MDT0001 complete [ 8005.964897] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 8008.524904] Key type lgssc unregistered [ 8008.845655] LNet: 266146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8008.864984] LNetError: 266146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8008.902466] LNet: Removed LNI 192.168.203.144@tcp [ 8009.737731] Key type .llcrypt unregistered [ 8009.748294] Key type ._llcrypt unregistered [ 8026.771405] Key type ._llcrypt registered [ 8026.773580] Key type .llcrypt registered [ 8026.878829] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8027.744731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8027.892917] alg: No test for adler32 (adler32-zlib) [ 8029.043699] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 8029.332301] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 8031.056226] Key type lgssc registered [ 8032.181594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8039.916590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8039.947485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8041.371995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8045.258456] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8051.685990] LustreError: 267368:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8051.715037] LustreError: 267368:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8052.568615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8052.935918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8057.278437] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8064.565161] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8069.882563] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8077.431703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8077.874177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8084.986211] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8086.023816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8092.441975] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8103.401292] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8103.418500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8103.440380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8106.466150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8106.467497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8106.476242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8106.517520] Lustre: Skipped 1 previous similar message [ 8109.060807] Lustre: server umount lustre-OST0000 complete [ 8114.656606] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8114.664676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8114.683026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8116.714475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8116.717972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8116.743697] Lustre: Skipped 1 previous similar message [ 8119.532990] Lustre: server umount lustre-MDT0000 complete [ 8121.824194] LustreError: 267368:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8121.841181] LustreError: 267368:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8123.099929] LustreError: 267348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123198 with bad export cookie 14489450311683009164 [ 8123.105039] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8123.106546] LustreError: 267348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8123.476317] Lustre: server umount lustre-MDT0001 complete [ 8137.390130] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 8140.659648] Key type lgssc unregistered [ 8141.019534] LNet: 271049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8141.030187] LNetError: 271049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8141.055750] LNet: Removed LNI 192.168.203.144@tcp [ 8142.094569] Key type .llcrypt unregistered [ 8142.096361] Key type ._llcrypt unregistered [ 8166.063312] Key type ._llcrypt registered [ 8166.068218] Key type .llcrypt registered [ 8166.213839] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8167.469722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8167.506217] alg: No test for adler32 (adler32-zlib) [ 8168.685922] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 8168.998918] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 8170.719200] Key type lgssc registered [ 8171.703790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8178.800565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8178.822200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8180.389500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8184.152243] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8190.949835] LustreError: 272465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8190.986139] LustreError: 272465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8192.461962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8192.914465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8197.617747] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8207.147820] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8213.807171] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8220.980489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8221.365955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8226.492096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8227.536350] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8235.784706] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8262.624940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8262.635557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8262.654076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8263.654234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8263.654548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8263.663777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8268.177145] Lustre: server umount lustre-OST0000 complete [ 8273.888905] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8273.892626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8273.902670] Lustre: Skipped 1 previous similar message [ 8273.913747] Lustre: Skipped 2 previous similar messages [ 8278.483351] Lustre: server umount lustre-MDT0000 complete [ 8279.011467] LustreError: 272460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8279.033449] LustreError: 272460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8282.149330] LustreError: 272447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123357 with bad export cookie 11987442262531068746 [ 8282.153948] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8282.161050] LustreError: 272447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8282.516665] Lustre: server umount lustre-MDT0001 complete [ 8294.484543] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 8297.624130] Key type lgssc unregistered [ 8297.996563] LNet: 276772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8298.001996] LNetError: 276772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8298.032318] LNet: Removed LNI 192.168.203.144@tcp [ 8298.955219] Key type .llcrypt unregistered [ 8298.957544] Key type ._llcrypt unregistered [ 8317.575436] Key type ._llcrypt registered [ 8317.576912] Key type .llcrypt registered [ 8317.772181] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8319.121823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8319.265274] alg: No test for adler32 (adler32-zlib) [ 8320.358849] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 8320.587058] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 8322.295163] Key type lgssc registered [ 8323.480953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8332.838273] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8332.878445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8334.479239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8339.235322] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8344.036469] LustreError: 278008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8344.056977] LustreError: 278008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8347.839659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8348.308235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8353.577480] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8362.158310] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8367.843239] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8374.648456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8375.079763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8379.197822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8381.773246] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8390.728269] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8409.057308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8409.063427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8409.072403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8409.091044] Lustre: Skipped 1 previous similar message [ 8412.718591] Lustre: server umount lustre-OST0000 complete [ 8419.297039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8419.312381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8419.323923] Lustre: Skipped 2 previous similar messages [ 8422.829880] Lustre: server umount lustre-MDT0000 complete [ 8424.416727] LustreError: 278009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8424.443277] LustreError: 278009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8426.868380] LustreError: 277974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123501 with bad export cookie 12691567843715343918 [ 8426.886954] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8427.260696] Lustre: server umount lustre-MDT0001 complete [ 8437.546125] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 8440.464199] Key type lgssc unregistered [ 8440.813594] LNet: 281711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8440.828314] LNetError: 281711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8440.854040] LNet: Removed LNI 192.168.203.144@tcp [ 8441.812385] Key type .llcrypt unregistered [ 8441.815240] Key type ._llcrypt unregistered [ 8453.864163] Key type ._llcrypt registered [ 8453.868329] Key type .llcrypt registered [ 8454.006317] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8473.952778] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 8492.720576] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8494.231453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8494.251237] alg: No test for adler32 (adler32-zlib) [ 8495.596579] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 8496.039604] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 8497.919978] Key type lgssc registered [ 8499.689540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8547.673772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8547.741343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8549.020927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8549.053407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8549.109718] Lustre: lustre-MDT0000: new disk, initializing [ 8549.195612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8549.212852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8555.189774] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8567.756768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8567.862924] Lustre: 285267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8567.905560] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8567.909892] Lustre: Skipped 1 previous similar message [ 8567.988192] Lustre: lustre-MDT0001: new disk, initializing [ 8568.064561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8568.120957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8568.134765] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8573.683661] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8584.095688] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8590.290617] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8597.412391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8597.775750] Lustre: lustre-OST0000: new disk, initializing [ 8597.782181] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8597.802615] Lustre: 287235:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8597.918703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8603.260330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8603.270664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8603.457301] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8606.095383] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8616.374479] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8621.551809] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8621.803955] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8626.031847] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8626.250377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8629.226174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8629.242133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8629.246955] LustreError: Skipped 1 previous similar message [ 8629.247776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8629.297783] Lustre: Skipped 1 previous similar message [ 8634.348437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8634.359749] Lustre: Skipped 2 previous similar messages [ 8634.817188] Lustre: server umount lustre-OST0000 complete [ 8639.458905] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8639.473052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8639.474890] Lustre: Skipped 1 previous similar message [ 8639.491226] Lustre: Skipped 1 previous similar message [ 8644.578330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8644.584013] Lustre: Skipped 1 previous similar message [ 8644.916901] Lustre: server umount lustre-MDT0000 complete [ 8648.927859] LustreError: 285260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123724 with bad export cookie 8922207544001617256 [ 8648.929190] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8648.939798] LustreError: 285260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8649.305652] Lustre: server umount lustre-MDT0001 complete [ 8658.160499] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:15:32 (1787123732) [ 8665.479614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8666.205304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8671.194461] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8671.212092] LustreError: 289650:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8671.231361] LustreError: 289650:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8676.326734] LustreError: 289651:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8679.971704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8680.389112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8685.308713] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8693.527657] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8699.595934] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8706.503693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8706.932768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8714.096331] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8722.182774] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8736.736824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8736.746668] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8736.765217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8740.843845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8740.844510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8742.408442] Lustre: server umount lustre-OST0000 complete [ 8746.986332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8746.994171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8747.011350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8747.023535] Lustre: Skipped 2 previous similar messages [ 8752.679642] Lustre: server umount lustre-MDT0000 complete [ 8756.192728] LustreError: 289651:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8756.218708] LustreError: 289651:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8756.693656] LustreError: 289632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123831 with bad export cookie 8922207544001618138 [ 8756.706472] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8757.181810] Lustre: server umount lustre-MDT0001 complete [ 8768.028821] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 8770.747846] Key type lgssc unregistered [ 8771.091103] LNet: 293209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8771.101768] LNetError: 293209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8771.117379] LNet: Removed LNI 192.168.203.144@tcp [ 8771.935169] Key type .llcrypt unregistered [ 8771.940515] Key type ._llcrypt unregistered [ 8799.467909] Key type ._llcrypt registered [ 8799.485803] Key type .llcrypt registered [ 8799.612250] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 8814.686569] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8815.747460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8815.808523] alg: No test for adler32 (adler32-zlib) [ 8816.957793] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 8817.229213] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 8818.895119] Key type lgssc registered [ 8820.168562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8869.506153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8869.549692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8870.780685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8870.829193] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8870.946280] Lustre: lustre-MDT0000: new disk, initializing [ 8871.063207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8871.086759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8876.320334] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8887.101749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8887.227610] Lustre: 297036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8887.291454] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8887.295801] Lustre: Skipped 1 previous similar message [ 8887.379916] Lustre: lustre-MDT0001: new disk, initializing [ 8887.495666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8887.559439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8887.585608] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8892.209056] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8901.382715] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8906.696727] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8913.924721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8914.197619] Lustre: lustre-OST2710: new disk, initializing [ 8914.202641] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8914.213542] Lustre: 299001:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8914.304023] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8921.669075] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8922.210302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8922.227627] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8922.365432] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8932.669792] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8939.486771] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8939.627801] Lustre: lustre-OST03e8: new disk, initializing [ 8939.633581] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8939.644125] Lustre: 300066:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8939.749690] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8946.959577] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8948.283115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8948.290949] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8948.368393] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8954.378451] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8963.369786] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8963.599614] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8970.687300] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8971.099761] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8976.397379] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8976.764854] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8980.185643] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8980.564804] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8985.069572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8985.079910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8985.097649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8986.775935] Lustre: server umount lustre-MDT0000 complete [ 8989.159629] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8989.180608] Lustre: Skipped 2 previous similar messages [ 8989.911600] LustreError: 297026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124064 with bad export cookie 17818547177256241506 [ 8989.912839] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8989.917923] LustreError: 297026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8990.196873] Lustre: server umount lustre-MDT0001 complete [ 9003.753176] Lustre: server umount lustre-OST2710 complete [ 9017.281897] Lustre: server umount lustre-OST03e8 complete [ 9037.326615] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 9048.283785] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9085.435811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9085.663837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9085.686142] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9085.785368] Lustre: lustre-MDT0000: new disk, initializing [ 9085.887312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9085.913483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9090.644952] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9101.861478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9102.017259] Lustre: 305318:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9102.073990] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9102.078345] Lustre: Skipped 1 previous similar message [ 9102.156439] Lustre: lustre-MDT0001: new disk, initializing [ 9102.254816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9102.312031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9102.327555] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9107.439312] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9117.833196] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9123.866904] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9132.480114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9132.926769] Lustre: lustre-OST0000: new disk, initializing [ 9132.946786] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9132.956047] Lustre: 307286:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9133.104839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9139.819883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9139.849431] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9139.979428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9140.668641] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9151.768981] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9156.976909] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9157.284606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9162.432886] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9162.689848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9165.795150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9165.802490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9165.825756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9165.847336] Lustre: Skipped 1 previous similar message [ 9165.863916] Lustre: Skipped 1 previous similar message [ 9170.933021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9170.940119] Lustre: Skipped 1 previous similar message [ 9171.267800] Lustre: server umount lustre-OST0000 complete [ 9176.032634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9176.038641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9176.047013] Lustre: Skipped 1 previous similar message [ 9176.073016] Lustre: Skipped 1 previous similar message [ 9181.153293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9181.170623] Lustre: Skipped 1 previous similar message [ 9190.367127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9190.533791] Lustre: server umount lustre-MDT0000 complete [ 9191.398419] LustreError: 305329:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9191.418412] LustreError: 305329:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9194.012438] LustreError: 305309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124269 with bad export cookie 17818547177256243186 [ 9194.019735] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9194.041046] LustreError: 305309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9194.296981] Lustre: server umount lustre-MDT0001 complete [ 9204.772208] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:24:38 (1787124278) [ 9206.746757] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9208.933450] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:24:42 (1787124282) [ 9213.097362] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9237.825870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9238.106271] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9238.126615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9238.257304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9241.886114] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9250.285871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9250.432572] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9250.462564] Lustre: 311145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9256.219638] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9280.578050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9280.883865] Lustre: Denying initial registration attempt from nid 192.168.203.144@tcp, specified as failover [ 9280.889310] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9280.895639] LustreError: 312600:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9280.900975] LustreError: 312600:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9280.905362] LustreError: 312600:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9280.960405] Lustre: server umount lustre-OST0000 complete [ 9280.966049] LustreError: 312600:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9291.232711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9291.234695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9291.244568] Lustre: Skipped 1 previous similar message [ 9291.270323] Lustre: Skipped 3 previous similar messages [ 9294.220256] Lustre: server umount lustre-MDT0000 complete [ 9296.358318] LustreError: 311154:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9296.394856] LustreError: 311154:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9298.274668] LustreError: 311137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124373 with bad export cookie 17818547177256244061 [ 9298.284192] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9298.291059] LustreError: 311137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9298.711747] Lustre: server umount lustre-MDT0001 complete [ 9309.174723] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 9311.914852] Key type lgssc unregistered [ 9312.226216] LNet: 313721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9312.235756] LNetError: 313721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9312.249990] LNet: Removed LNI 192.168.203.144@tcp [ 9313.391124] Key type .llcrypt unregistered [ 9313.392835] Key type ._llcrypt unregistered [ 9327.382559] Key type ._llcrypt registered [ 9327.385585] Key type .llcrypt registered [ 9327.499532] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9328.583843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9328.615788] alg: No test for adler32 (adler32-zlib) [ 9329.644456] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 9329.921150] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 9331.559168] Key type lgssc registered [ 9332.615231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9376.788516] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9386.206561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9386.239564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9387.678612] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9387.732251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9388.021467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9393.186257] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9402.189941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9402.323732] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9402.378229] Lustre: 316490:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9402.724190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9408.874463] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9417.112214] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9423.729437] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9444.525413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9444.758548] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9445.055812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9451.805805] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9459.847371] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9462.768114] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9462.784819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9462.819105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9463.270282] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9463.286054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9468.385446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9468.397515] Lustre: Skipped 3 previous similar messages [ 9473.511147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9476.063242] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9476.248760] Lustre: server umount lustre-OST0000 complete [ 9483.748839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9483.777152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9483.789502] Lustre: Skipped 2 previous similar messages [ 9486.170597] Lustre: server umount lustre-MDT0000 complete [ 9489.825049] LustreError: 316481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124564 with bad export cookie 15149710265256559756 [ 9489.838442] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9489.840038] LustreError: 316481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9490.249755] Lustre: server umount lustre-MDT0001 complete [ 9502.581913] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 9505.539096] Key type lgssc unregistered [ 9505.853991] LNet: 320066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9505.863996] LNetError: 320066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9505.881803] LNet: Removed LNI 192.168.203.144@tcp [ 9506.795253] Key type .llcrypt unregistered [ 9506.798962] Key type ._llcrypt unregistered [ 9529.130503] Key type ._llcrypt registered [ 9529.132701] Key type .llcrypt registered [ 9529.225179] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9530.197424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9530.220748] alg: No test for adler32 (adler32-zlib) [ 9531.371098] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 9531.618804] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 9533.303175] Key type lgssc registered [ 9534.617670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9542.163179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9542.198544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9543.704132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9548.351025] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9553.900727] LustreError: 321462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9553.931280] LustreError: 321462:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9556.420521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9556.726224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9561.630216] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9569.383642] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9575.299914] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9583.918727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9584.243640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9590.127400] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9597.528869] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9612.262465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9612.269455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9612.296565] Lustre: Skipped 1 previous similar message [ 9614.949799] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.44@tcp (stopping) [ 9614.961161] Lustre: Skipped 2 previous similar messages [ 9617.379563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9617.389980] Lustre: Skipped 2 previous similar messages [ 9620.050266] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.44@tcp (stopping) [ 9625.164538] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.44@tcp (stopping) [ 9625.180656] Lustre: Skipped 3 previous similar messages [ 9626.079259] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9626.414194] Lustre: server umount lustre-MDT0000 complete [ 9627.619953] LustreError: 321457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9627.660278] LustreError: 321457:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9630.295116] LustreError: 322465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9632.216301] LustreError: 321442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124707 with bad export cookie 1779006359056622825 [ 9632.239243] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9632.743552] LustreError: 321457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9632.745481] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9632.785019] Lustre: Skipped 1 previous similar message [ 9635.482205] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.44@tcp (stopping) [ 9635.492262] Lustre: Skipped 1 previous similar message [ 9637.867136] LustreError: 322465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9637.885906] LustreError: 322465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9638.751496] Lustre: server umount lustre-MDT0001 complete [ 9646.543814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9655.935811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9657.183123] LustreError: 325065:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 9657.196645] LustreError: 325065:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@0000000094f4e8dd x1873935970901504/t0(0) o250->MGC192.168.203.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1787124732 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9657.238210] LustreError: 325065:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 9657.695699] LustreError: 323294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9657.820112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9657.910922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9663.310199] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9667.053449] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9673.487773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9673.955614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9678.732674] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9679.362518] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9687.431308] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9688.968967] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9694.788695] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9696.337657] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9703.905630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9703.916983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9703.930531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9703.941495] Lustre: Skipped 1 previous similar message [ 9704.937321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9707.377677] Lustre: server umount lustre-OST0000 complete [ 9715.169037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9715.178561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9715.199839] Lustre: Skipped 1 previous similar message [ 9716.981378] Lustre: server umount lustre-MDT0000 complete [ 9720.289369] LustreError: 325078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9720.304879] LustreError: 325078:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 9720.438979] LustreError: 321443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124795 with bad export cookie 1779006359056630301 [ 9720.444249] LustreError: 321443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9720.445322] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9720.677752] Lustre: server umount lustre-MDT0001 complete [ 9731.969289] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 9734.728462] Key type lgssc unregistered [ 9734.999514] LNet: 327752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9735.011139] LNetError: 327752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9735.031671] LNet: Removed LNI 192.168.203.144@tcp [ 9735.952326] Key type .llcrypt unregistered [ 9735.962857] Key type ._llcrypt unregistered [ 9758.649276] Key type ._llcrypt registered [ 9758.653919] Key type .llcrypt registered [ 9758.766506] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9759.948199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9759.965481] alg: No test for adler32 (adler32-zlib) [ 9761.111466] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 9761.399512] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 9763.080891] Key type lgssc registered [ 9764.519719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9772.209347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9772.251311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9773.663164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9778.233292] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9783.784471] LustreError: 329171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9783.806222] LustreError: 329171:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9786.465429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9786.840079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9790.998476] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9798.013214] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9802.540302] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9807.331755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9807.337102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9807.352061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9808.354933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9808.355244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9811.406624] Lustre: server umount lustre-MDT0000 complete [ 9811.489720] LustreError: 329165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9811.521908] LustreError: 329165:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9815.000359] LustreError: 329151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124890 with bad export cookie 1657700470679546105 [ 9815.002681] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9815.010281] LustreError: 329151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9815.349109] Lustre: server umount lustre-MDT0001 complete [ 9828.607447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9828.902846] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9828.946977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9829.163531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9833.047394] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9840.688644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9840.771044] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9840.834074] Lustre: 331757:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9841.093516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9844.725120] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9851.659258] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9856.989759] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9868.222554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9868.475816] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9868.495850] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9868.512294] LustreError: 333644:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9868.532780] LustreError: 333644:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9868.537299] LustreError: 333644:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9868.607532] Lustre: server umount lustre-OST0000 complete [ 9868.612223] LustreError: 333644:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9878.636531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9878.783545] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9878.963971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9885.358630] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9887.511489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9894.920842] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9903.817816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9903.914155] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9904.032493] Lustre: lustre-OST0001: new disk, initializing [ 9904.045861] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9904.055934] Lustre: 335183:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9904.193264] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9910.610564] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9913.976965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9913.989487] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9914.044332] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9918.548164] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9924.576459] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9924.592066] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9924.618925] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9924.631150] Lustre: Skipped 2 previous similar messages [ 9929.703668] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9929.716394] Lustre: Skipped 1 previous similar message [ 9934.828439] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9934.834928] Lustre: Skipped 1 previous similar message [ 9936.351129] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9936.457395] Lustre: server umount lustre-OST0001 complete [ 9939.958415] LustreError: 334109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9940.003134] LustreError: 334109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9944.032256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9944.035824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9944.041730] LustreError: Skipped 2 previous similar messages [ 9944.043312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9944.068296] Lustre: Skipped 2 previous similar messages [ 9945.083777] LustreError: 336179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9945.114949] LustreError: 336179:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9946.387859] Lustre: server umount lustre-OST0000 complete [ 9950.180063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9956.368376] Lustre: server umount lustre-MDT0000 complete [ 9960.288589] LustreError: 331751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125035 with bad export cookie 1657700470679546658 [ 9960.300458] LustreError: 331751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9960.307688] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9960.745684] Lustre: server umount lustre-MDT0001 complete [ 9970.974096] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 9973.493957] Key type lgssc unregistered [ 9973.785852] LNet: 337147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9973.792251] LNetError: 337147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9973.805517] LNet: Removed LNI 192.168.203.144@tcp [ 9974.447906] Key type .llcrypt unregistered [ 9974.449921] Key type ._llcrypt unregistered [10045.114393] Key type ._llcrypt registered [10045.116184] Key type .llcrypt registered [10045.165805] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [10061.033184] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10062.159242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10062.203700] alg: No test for adler32 (adler32-zlib) [10063.168467] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [10063.399714] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [10065.088965] Key type lgssc registered [10066.515297] Lustre: Echo OBD driver; http://www.lustre.org/ [10103.009178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10103.050765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10104.420769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10104.454747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10104.514351] Lustre: lustre-MDT0000: new disk, initializing [10104.632700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10104.645317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10108.650195] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10119.256863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10119.387836] Lustre: 342083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10119.471838] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10119.477279] Lustre: Skipped 1 previous similar message [10119.548908] Lustre: lustre-MDT0001: new disk, initializing [10119.650771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10119.684142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10119.698131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10123.195859] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10131.935955] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10137.946805] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10144.020256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10144.325709] Lustre: lustre-OST0000: new disk, initializing [10144.330518] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10144.336217] Lustre: 344048:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10144.424473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10150.493108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10150.500957] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10150.505105] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10150.621379] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10159.233975] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10163.001889] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10163.149386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10167.028892] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10167.223382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10169.316130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10169.320656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10169.330888] Lustre: Skipped 1 previous similar message [10174.433236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10174.440382] Lustre: Skipped 2 previous similar messages [10175.193132] Lustre: server umount lustre-OST0000 complete [10181.089751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10181.102509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10181.142868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10184.688742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10184.690048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10186.295306] Lustre: server umount lustre-MDT0000 complete [10189.792937] LustreError: 342094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10189.829791] LustreError: 342094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10191.600580] LustreError: 342075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125266 with bad export cookie 4382872927226557591 [10191.601631] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10191.607900] LustreError: 342075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10191.984533] Lustre: server umount lustre-MDT0001 complete [10203.230221] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 03:41:17 (1787125277) [10211.966387] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 03:41:26 (1787125286) [10221.897025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10222.398268] LustreError: 346803:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10222.534947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10227.326923] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10227.684873] LustreError: 346804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10232.801673] LustreError: 346803:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10235.980270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10236.394993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10240.880390] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10248.284167] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10254.492706] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10260.961098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10261.268784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10267.606382] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10275.495181] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10283.893684] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10285.982401] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10287.658178] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10289.496241] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10291.303121] Lustre: Failing over lustre-MDT0000 [10291.588675] Lustre: server umount lustre-MDT0000 complete [10292.704646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10292.712793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10292.721951] LustreError: 347532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10303.459200] LustreError: 348239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10303.494686] LustreError: 348239:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [10309.600129] Lustre: 340307:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787125368/real 1787125368] req@00000000ac6229e8 x1873936528771712/t0(0) o400->MGC192.168.203.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787125384 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10309.633964] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10310.758550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10319.844399] LustreError: 346803:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10319.848509] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3cd317b4b303fc1c [10319.862579] LustreError: 346803:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [10319.886836] LustreError: 350010:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.203.144@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000007d7fe5ca) refcount nonzero (2) after lock cleanup; forcing cleanup. [10319.908371] Lustre: MGC192.168.203.144@tcp: Connection restored to 0@lo (at 0@lo) [10319.949665] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.44@tcp (not set up) [10319.961968] Lustre: Skipped 1 previous similar message [10320.270711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10320.370984] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10325.066390] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10325.494282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10325.552483] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10325.622353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10326.078888] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10336.714879] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10338.759600] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10342.219051] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10344.172638] Lustre: Failing over lustre-MDT0000 [10344.510314] Lustre: server umount lustre-MDT0000 complete [10345.953338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10345.958111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10345.990810] Lustre: Skipped 4 previous similar messages [10355.788770] LustreError: 346798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10355.818839] LustreError: 346798:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [10362.341196] Lustre: 340305:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787125421/real 1787125421] req@00000000f6da78fd x1873936528793600/t0(0) o400->MGC192.168.203.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787125437 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10362.363310] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10363.770135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10372.576138] LustreError: 340304:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ee2c979e x1873936528799232/t0(0) o250->MGC192.168.203.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10372.919159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10372.975175] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10377.296558] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10377.415685] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10378.224057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10378.234699] Lustre: Skipped 2 previous similar messages [10378.285376] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10378.344720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10387.241493] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10389.191780] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10398.693537] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10398.695405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10398.698860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10398.713923] Lustre: Skipped 1 previous similar message [10401.932535] Lustre: server umount lustre-MDT0000 complete [10406.109269] LustreError: 346785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125481 with bad export cookie 4382872927226560314 [10406.117784] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10406.132087] LustreError: 346785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10406.673307] Lustre: server umount lustre-MDT0001 complete [10421.641737] Lustre: server umount lustre-OST0000 complete [10436.194530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10436.693611] LustreError: 353146:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10436.727776] LustreError: 353146:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 22 previous similar messages [10441.387474] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10450.290185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10450.873900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10450.877323] Lustre: Skipped 1 previous similar message [10455.578877] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10462.318081] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10467.435756] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10474.869390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10481.590494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10482.310233] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10490.188823] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10497.011746] Lustre: server umount lustre-OST0000 complete [10502.114875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10502.115327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10502.125753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10502.133802] Lustre: Skipped 2 previous similar messages [10502.144379] Lustre: Skipped 3 previous similar messages [10511.403059] Lustre: server umount lustre-MDT0000 complete [10513.375152] Lustre: 340305:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787125572/real 1787125572] req@00000000555f1428 x1873936528844544/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787125588 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10514.952784] LustreError: 353126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125590 with bad export cookie 4382872927226561049 [10514.955553] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10514.961750] LustreError: 353126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10515.340811] Lustre: server umount lustre-MDT0001 complete [10527.565289] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [10530.413980] Key type lgssc unregistered [10530.687751] LNet: 356680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10530.695513] LNetError: 356680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10530.707274] LNet: Removed LNI 192.168.203.144@tcp [10531.609279] Key type .llcrypt unregistered [10531.615369] Key type ._llcrypt unregistered [10554.792766] Key type ._llcrypt registered [10554.799605] Key type .llcrypt registered [10554.916116] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10555.825975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10555.935647] alg: No test for adler32 (adler32-zlib) [10557.017235] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [10557.282583] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [10558.999164] Key type lgssc registered [10560.124598] Lustre: Echo OBD driver; http://www.lustre.org/ [10567.433126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10567.446167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10568.865521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10573.363893] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10578.410447] LustreError: 358178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10578.424553] LustreError: 358178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10581.856452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10582.282245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10586.336456] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10593.251745] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10598.794515] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10605.467825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10605.881119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10612.869164] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10613.050177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10624.128723] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10637.430755] Lustre: server umount lustre-MDT0000 complete [10638.823819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10638.824267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10638.841124] LustreError: 359233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10638.865525] LustreError: 359233:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10640.855474] LustreError: 358159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125715 with bad export cookie 2253282640644655263 [10640.871272] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10641.233767] Lustre: server umount lustre-MDT0001 complete [10655.515160] Lustre: server umount lustre-OST0000 complete [10669.354281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10669.836616] LustreError: 361890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10669.885705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10674.632320] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10675.207399] LustreError: 361891:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10679.329944] LustreError: 361890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10685.530261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10686.088567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10690.734481] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10698.568437] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10705.157365] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10712.612463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10712.903593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10713.966478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10718.955740] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10726.128332] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10741.733756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10741.756282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10741.760037] Lustre: Skipped 3 previous similar messages [10741.780134] Lustre: Skipped 2 previous similar messages [10743.732071] Lustre: server umount lustre-MDT0000 complete [10746.850550] LustreError: 362624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10746.881402] LustreError: 362624:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10747.287909] LustreError: 361870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125822 with bad export cookie 2253282640644656376 [10747.292932] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10747.320512] LustreError: 361870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10747.723047] Lustre: server umount lustre-MDT0001 complete [10762.106725] Lustre: server umount lustre-OST0000 complete [10777.233693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10777.706263] LustreError: 365802:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10777.794283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10783.176478] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10792.264267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10792.416135] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10797.249445] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10805.172099] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10810.711505] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10817.351134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10817.511832] LustreError: 367622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10817.533528] LustreError: 367622:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10817.661752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10817.668533] Lustre: Skipped 1 previous similar message [10822.786297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10825.433462] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10834.142423] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10843.623317] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10843.640344] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10843.643066] LustreError: Skipped 1 previous similar message [10843.671712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10843.699571] Lustre: Skipped 2 previous similar messages [10848.746306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10848.753473] Lustre: Skipped 2 previous similar messages [10853.869163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10853.875309] Lustre: Skipped 1 previous similar message [10855.391125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10855.506935] Lustre: server umount lustre-OST0000 complete [10858.978619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10858.988918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10859.008065] Lustre: Skipped 1 previous similar message [10864.096744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10864.104170] Lustre: Skipped 3 previous similar messages [10873.311117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10873.507889] Lustre: server umount lustre-MDT0000 complete [10877.250874] LustreError: 365784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125952 with bad export cookie 2253282640644657398 [10877.270630] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10877.577433] Lustre: server umount lustre-MDT0001 complete [10888.459687] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [10891.333327] Key type lgssc unregistered [10891.639589] LNet: 369339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10891.659932] LNetError: 369339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10891.687875] LNet: Removed LNI 192.168.203.144@tcp [10892.726782] Key type .llcrypt unregistered [10892.736606] Key type ._llcrypt unregistered [10918.833734] Key type ._llcrypt registered [10918.838274] Key type .llcrypt registered [10918.972159] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10920.192070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10920.245094] alg: No test for adler32 (adler32-zlib) [10921.375482] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [10921.601431] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [10923.287163] Key type lgssc registered [10924.127095] Lustre: Echo OBD driver; http://www.lustre.org/ [10931.086793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10931.103884] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10931.109863] LustreError: 370780:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10931.124906] LustreError: 370780:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10931.131547] LustreError: 370780:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10931.136532] LustreError: 370780:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10931.141927] LustreError: 370780:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [10938.597238] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10938.615173] LustreError: 371178:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10938.641053] LustreError: 371178:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10938.651692] LustreError: 371178:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10938.658476] LustreError: 371178:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10938.666877] LustreError: 371178:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [10967.414495] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [10971.087602] Key type lgssc unregistered [10971.384416] LNet: 372203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10971.389494] LNetError: 372203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10971.402597] LNet: Removed LNI 192.168.203.144@tcp [10972.260933] Key type .llcrypt unregistered [10972.264183] Key type ._llcrypt unregistered [10990.566691] Key type ._llcrypt registered [10990.573138] Key type .llcrypt registered [10990.732398] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [11006.538587] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [11007.428772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11007.539196] alg: No test for adler32 (adler32-zlib) [11008.634448] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [11008.894525] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [11010.551176] Key type lgssc registered [11011.725645] Lustre: Echo OBD driver; http://www.lustre.org/ [11049.069318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11049.101245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11050.386544] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11050.434876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11050.548083] Lustre: lustre-MDT0000: new disk, initializing [11050.661589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11050.692637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11055.235969] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11064.981550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11065.130940] Lustre: 375360:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11065.185619] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11065.195323] Lustre: Skipped 1 previous similar message [11065.294615] Lustre: lustre-MDT0001: new disk, initializing [11065.407797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11065.466179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11065.487743] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11070.079837] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11082.626490] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11088.924879] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11094.866532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11095.125415] Lustre: lustre-OST0000: new disk, initializing [11095.131291] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11095.141190] Lustre: 377329:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11095.245902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11101.229136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11101.250113] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11101.362913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11101.620624] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11111.616578] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11116.414033] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11116.722287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11121.988833] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11122.315076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11127.265924] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11127.268987] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11127.285761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11127.293107] Lustre: Skipped 1 previous similar message [11130.886376] Lustre: server umount lustre-OST0000 complete [11137.000465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11137.015545] LustreError: Skipped 1 previous similar message [11137.021312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11137.037932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11137.051557] Lustre: Skipped 1 previous similar message [11140.981888] Lustre: server umount lustre-MDT0000 complete [11145.333793] LustreError: 375352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126220 with bad export cookie 17381015544295914696 [11145.334509] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11145.344052] LustreError: 375352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11145.657108] Lustre: server umount lustre-MDT0001 complete [11154.009746] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11155.934730] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 03:57:09 (1787126229) [11162.542909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11163.068240] LustreError: 379787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11163.099256] LustreError: 379787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11163.140535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11168.231774] LustreError: 379788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11168.263495] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11173.350334] LustreError: 379787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11177.057572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11177.354513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11181.312607] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11189.437724] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11197.298061] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11206.598240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11206.907199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11214.144130] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11221.893247] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11228.455646] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11228.573580] Lustre: lustre-OST0001: new disk, initializing [11228.576469] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11228.582903] Lustre: 382625:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11228.657252] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11234.714765] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11236.484594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11236.502619] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11236.602984] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11246.141320] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11252.210220] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11252.217679] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11252.244715] LustreError: Skipped 1 previous similar message [11252.251240] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11252.272852] Lustre: Skipped 2 previous similar messages [11252.288555] Lustre: Skipped 3 previous similar messages [11256.445325] Lustre: server umount lustre-OST0001 complete [11257.355940] LustreError: 381611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11257.379338] LustreError: 381611:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11262.439119] LustreError: 381608:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11262.463276] LustreError: 381608:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11272.680523] LustreError: 381608:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11272.717973] LustreError: 381608:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11275.744766] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11275.757428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11275.793539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11275.806476] Lustre: Skipped 1 previous similar message [11280.353819] Lustre: server umount lustre-OST0000 complete [11288.036802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11288.062058] Lustre: Skipped 1 previous similar message [11288.077353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11288.080392] Lustre: Skipped 2 previous similar messages [11291.374364] Lustre: server umount lustre-MDT0000 complete [11293.157173] LustreError: 379783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11293.175656] LustreError: 379783:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11296.075058] LustreError: 382636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126371 with bad export cookie 17381015544295915578 [11296.077240] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11296.086210] LustreError: 382636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11302.599524] Lustre: server umount lustre-MDT0001 complete [11314.176598] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [11316.875849] Key type lgssc unregistered [11317.156741] LNet: 384611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11317.166648] LNetError: 384611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11317.183749] LNet: Removed LNI 192.168.203.144@tcp [11318.058904] Key type .llcrypt unregistered [11318.061686] Key type ._llcrypt unregistered [11366.672134] Key type ._llcrypt registered [11366.674290] Key type .llcrypt registered [11366.775281] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [11367.765375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11367.926821] alg: No test for adler32 (adler32-zlib) [11369.103792] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [11369.350286] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [11371.032302] Key type lgssc registered [11372.332317] Lustre: Echo OBD driver; http://www.lustre.org/ [11379.453884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11379.471982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11380.785804] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11380.820736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11380.970521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11385.670241] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11395.274248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11395.433960] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11395.479300] Lustre: 387408:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11395.890843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11400.403284] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11407.981880] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11413.393156] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11416.546393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11416.559150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11416.572317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11418.082075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11418.083427] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11421.689876] Lustre: server umount lustre-MDT0000 complete [11424.793904] LustreError: 387403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126499 with bad export cookie 10517979415424305759 [11424.802877] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11424.813407] LustreError: 387403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11425.072250] Lustre: server umount lustre-MDT0001 complete [11435.029749] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 04:01:48 (1787126508) [11441.675397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11442.111372] LustreError: 389931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11442.126278] LustreError: 389931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11442.222100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11446.533454] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11447.267047] LustreError: 389932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11452.384821] LustreError: 389931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11455.137677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11455.574093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11460.178850] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11468.104798] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11473.956972] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11481.876451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11482.085303] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11482.291677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11487.792427] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11495.030450] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11503.340795] Lustre: Permanently deactivating lustre-OST0000 [11503.366212] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11508.780472] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11510.664568] LustreError: 392871:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11510.673657] LustreError: 392871:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11510.679038] LustreError: 392871:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11513.780704] LustreError: 392919:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11513.786430] LustreError: 392919:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11521.818531] Lustre: server umount lustre-OST0000 complete [11524.413924] LustreError: 393169:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11524.418609] LustreError: 393169:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11527.137604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11527.145112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11527.170137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11527.178689] Lustre: Skipped 1 previous similar message [11533.283451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11533.302517] Lustre: Skipped 3 previous similar messages [11536.426972] Lustre: server umount lustre-MDT0000 complete [11538.409908] LustreError: 391466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11538.432616] LustreError: 391466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11540.193989] LustreError: 389913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126615 with bad export cookie 10517979415424306340 [11540.196868] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11540.202439] LustreError: 389913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11550.858160] Lustre: server umount lustre-MDT0001 complete [11559.574394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11564.292473] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11567.055404] LustreError: 394329:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11573.920216] Lustre: 394670:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.144@tcp [11576.494650] Lustre: server umount MGS complete [11589.203839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11589.671426] LustreError: 395394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11589.704761] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11589.715430] Lustre: Skipped 1 previous similar message [11589.788880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11594.648660] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11600.356205] LustreError: 395394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11600.381608] LustreError: 395394:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11603.171034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11603.434642] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11608.071908] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11616.861349] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11623.267414] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11625.124966] Lustre: Permanently reactivating lustre-OST0000 [11625.147493] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11625.155968] Lustre: Skipped 2 previous similar messages [11652.579384] LustreError: 397174:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11661.063568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11661.495516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11661.501710] Lustre: Skipped 1 previous similar message [11668.328450] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11677.371646] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11683.296386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11683.299602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11683.307050] LustreError: Skipped 1 previous similar message [11683.310666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11683.343791] Lustre: Skipped 2 previous similar messages [11689.543142] Lustre: server umount lustre-OST0000 complete [11693.536103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11693.549094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11693.550355] Lustre: Skipped 1 previous similar message [11693.579035] Lustre: Skipped 4 previous similar messages [11699.882768] Lustre: server umount lustre-MDT0000 complete [11703.777153] LustreError: 395390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11703.804500] LustreError: 395390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11704.206454] LustreError: 395374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126779 with bad export cookie 10517979415424307390 [11704.208916] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11704.227474] LustreError: 395374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11704.683326] Lustre: server umount lustre-MDT0001 complete [11717.114572] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [11720.388974] Key type lgssc unregistered [11720.757812] LNet: 399805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11720.770579] LNetError: 399805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11720.796476] LNet: Removed LNI 192.168.203.144@tcp [11721.964306] Key type .llcrypt unregistered [11721.966871] Key type ._llcrypt unregistered [11743.616257] Key type ._llcrypt registered [11743.617762] Key type .llcrypt registered [11743.702971] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [11759.331985] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [11760.698838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11760.743918] alg: No test for adler32 (adler32-zlib) [11761.991650] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [11762.301578] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [11763.999535] Key type lgssc registered [11765.613273] Lustre: Echo OBD driver; http://www.lustre.org/ [11803.474917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11803.522569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11804.848552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11804.900145] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11805.039220] Lustre: lustre-MDT0000: new disk, initializing [11805.153439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11805.178079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11809.360413] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11820.707546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11820.801553] Lustre: 402963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11820.857549] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11820.863468] Lustre: Skipped 1 previous similar message [11820.940036] Lustre: lustre-MDT0001: new disk, initializing [11821.046901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11821.083729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11821.096630] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11825.720644] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11835.249886] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11840.908283] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11848.458815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11848.644279] Lustre: lustre-OST0000: new disk, initializing [11848.646980] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11848.656234] Lustre: 404932:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11848.741521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11855.011450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11855.047786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11855.249752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11857.877538] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11868.387115] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11872.594460] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11872.916802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11876.616771] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11876.755113] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11880.930283] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11880.935509] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11880.953070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11880.963050] Lustre: Skipped 1 previous similar message [11884.678195] Lustre: server umount lustre-OST0000 complete [11891.172045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11891.182841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11891.208703] Lustre: Skipped 2 previous similar messages [11891.214665] Lustre: Skipped 1 previous similar message [11894.826097] Lustre: server umount lustre-MDT0000 complete [11898.826564] LustreError: 402955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126973 with bad export cookie 3475996402413002777 [11898.830140] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11898.837029] LustreError: 402955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11899.299660] Lustre: server umount lustre-MDT0001 complete [11908.453515] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11910.290744] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 04:09:44 (1787126984) [11917.418688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11917.895186] LustreError: 407395:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11917.917570] LustreError: 407395:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11918.007033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11922.938896] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11923.435180] LustreError: 407396:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11928.546493] LustreError: 407395:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11931.795288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11932.249570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11937.195930] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11944.678927] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11950.419858] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11956.680459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11957.041255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11963.839603] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11971.808750] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11973.688643] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11976.254365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11983.847180] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11983.854702] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11983.858207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11983.881820] Lustre: Skipped 1 previous similar message [11988.900551] Lustre: server umount lustre-OST0000 complete [11993.568230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11993.578811] LustreError: Skipped 1 previous similar message [11993.582323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11993.594488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11993.607823] Lustre: Skipped 1 previous similar message [11999.201980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11999.215272] Lustre: Skipped 3 previous similar messages [12007.391115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12007.707550] Lustre: server umount lustre-MDT0000 complete [12009.450128] LustreError: 407395:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12009.486541] LustreError: 407395:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12012.977204] LustreError: 407376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127088 with bad export cookie 3475996402413003666 [12012.982513] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12013.002551] LustreError: 407376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12014.380923] Lustre: server umount lustre-MDT0001 complete [12025.432854] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12028.535837] Key type lgssc unregistered [12028.852895] LNet: 411124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12028.865220] LNetError: 411124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12028.891434] LNet: Removed LNI 192.168.203.144@tcp [12029.934547] Key type .llcrypt unregistered [12029.936655] Key type ._llcrypt unregistered [12053.478580] Key type ._llcrypt registered [12053.480740] Key type .llcrypt registered [12053.619744] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12053.926392] Key type .llcrypt unregistered [12053.933306] Key type ._llcrypt unregistered [12071.964957] Key type ._llcrypt registered [12071.968619] Key type .llcrypt registered [12072.079943] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [12073.145386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12073.220455] alg: No test for adler32 (adler32-zlib) [12074.372470] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [12074.658830] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [12076.335173] Key type lgssc registered [12077.683412] Lustre: Echo OBD driver; http://www.lustre.org/ [12085.166601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12085.193248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12086.728806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12091.625461] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12095.974582] LustreError: 413336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12096.003647] LustreError: 413336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12100.871983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12101.094045] LustreError: 413336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12101.478624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12106.721309] LustreError: 413337:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12106.756520] LustreError: 413337:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12109.322334] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12111.841958] LustreError: 413336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12111.864383] LustreError: 413336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12111.869113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12116.969591] LustreError: 413333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12117.009422] LustreError: 413333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12118.515204] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12125.453082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12125.721402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12129.665608] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12140.137047] Lustre: server umount lustre-OST0000 complete [12146.148779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12146.155035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12146.159029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12146.173764] Lustre: Skipped 1 previous similar message [12151.274636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12151.286295] Lustre: Skipped 2 previous similar messages [12154.427296] Lustre: server umount lustre-MDT0000 complete [12156.388875] LustreError: 415073:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12156.408140] LustreError: 415073:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12157.407369] Lustre: 412650:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787127216/real 1787127216] req@0000000075249096 x1873938637417216/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787127232 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12157.410454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12157.464148] Lustre: 412650:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [12158.971240] LustreError: 415443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127234 with bad export cookie 14965249118460202321 [12158.981904] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12159.003052] LustreError: 415443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12159.490173] Lustre: server umount lustre-MDT0001 complete [12170.084242] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12172.367266] Key type lgssc unregistered [12172.607611] LNet: 416484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12172.617587] LNetError: 416484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12172.637716] LNet: Removed LNI 192.168.203.144@tcp [12173.423625] Key type .llcrypt unregistered [12173.428939] Key type ._llcrypt unregistered [12198.162570] Key type ._llcrypt registered [12198.164364] Key type .llcrypt registered [12198.299628] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [12199.217252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12199.240624] alg: No test for adler32 (adler32-zlib) [12200.400289] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [12200.688770] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [12202.471148] Key type lgssc registered [12203.878568] Lustre: Echo OBD driver; http://www.lustre.org/ [12212.240346] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12212.289479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12227.807473] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12232.928374] LustreError: 417878:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.144@tcp: failed processing log, type 4: rc = -110 [12259.615193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12265.109430] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12272.057758] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12279.506137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12280.121534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12280.195607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12284.391086] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12285.408475] LustreError: 417903:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12285.454180] LustreError: 417903:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12290.542218] LustreError: 417904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12293.133925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12293.628277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12298.443345] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12305.769440] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12311.202432] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12320.226398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12320.230941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12320.255600] Lustre: Skipped 1 previous similar message [12320.267989] Lustre: Skipped 1 previous similar message [12321.921186] Lustre: server umount lustre-OST0000 complete [12325.894212] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12325.907541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12325.919316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12326.094454] Lustre: server umount lustre-MDT0000 complete [12329.857699] LustreError: 417884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127404 with bad export cookie 5774898220625126939 [12329.863760] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12329.885969] LustreError: 417884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12330.205963] Lustre: server umount lustre-MDT0001 complete [12342.285305] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12344.988774] Key type lgssc unregistered [12345.288635] LNet: 421445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12345.298525] LNetError: 421445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12345.316613] LNet: Removed LNI 192.168.203.144@tcp [12346.208954] Key type .llcrypt unregistered [12346.212233] Key type ._llcrypt unregistered [12375.364992] Key type ._llcrypt registered [12375.371981] Key type .llcrypt registered [12375.494846] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [12376.688323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12376.716832] alg: No test for adler32 (adler32-zlib) [12377.734637] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [12377.892707] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [12379.615674] Key type lgssc registered [12381.000640] Lustre: Echo OBD driver; http://www.lustre.org/ [12389.464839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12389.501714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12391.101861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12395.586084] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12401.636679] LustreError: 422864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12401.667992] LustreError: 422864:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12404.697400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12405.106613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12410.209072] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12418.493284] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12425.128526] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12432.709615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12433.057531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12436.152353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12440.552564] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12449.861516] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12455.909924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12455.922803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12455.947753] Lustre: Skipped 1 previous similar message [12455.978881] Lustre: Skipped 2 previous similar messages [12457.283693] Lustre: server umount lustre-MDT0000 complete [12457.587043] LustreError: 422858:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12459.955570] LustreError: 422845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.44@tcp arrived at 1787127535 with bad export cookie 5508637667315088897 [12459.989135] LustreError: 422845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12461.030923] LustreError: 423851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12461.060534] LustreError: 423851:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12461.981491] LustreError: 422843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127537 with bad export cookie 5508637667315088001 [12461.997274] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12462.000071] LustreError: 422843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12466.154848] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12466.157579] LustreError: 423591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12466.168321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12466.183985] LustreError: 423591:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12466.194856] Lustre: Skipped 1 previous similar message [12466.205840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12468.212246] Lustre: server umount lustre-OST0000 complete [12471.279736] LustreError: 423851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12471.320609] LustreError: 423851:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12475.237281] Lustre: server umount lustre-MDT0001 complete [12487.864377] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12490.616975] Key type lgssc unregistered [12490.994960] LNet: 426548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12491.000511] LNetError: 426548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12492.078709] LNet: Removed LNI 192.168.203.144@tcp [12492.967624] Key type .llcrypt unregistered [12492.970112] Key type ._llcrypt unregistered [12519.937562] Key type ._llcrypt registered [12519.939795] Key type .llcrypt registered [12520.090556] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [12521.306682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12521.359519] alg: No test for adler32 (adler32-zlib) [12522.454479] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [12522.786806] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [12524.511159] Key type lgssc registered [12525.652850] Lustre: Echo OBD driver; http://www.lustre.org/ [12533.753955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12533.774953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12535.240258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12539.953431] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12545.507843] LustreError: 427969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12545.535705] LustreError: 427969:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12549.165968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12549.492445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12554.175164] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12562.697557] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12568.586416] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12575.344680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12575.866678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12579.971429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12582.526443] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12590.963871] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12595.684290] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12595.698355] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12595.711864] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12598.866866] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.44@tcp (stopping) [12599.796335] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12601.958849] Lustre: server umount lustre-MDT0001 complete [12604.898330] LustreError: 427969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12604.944489] LustreError: 427969:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12610.019741] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12610.020213] LustreError: 427968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12610.038605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12610.070450] Lustre: Skipped 2 previous similar messages [12614.420087] Lustre: server umount lustre-OST0000 complete [12615.137126] LustreError: 427969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12618.689532] Lustre: server umount lustre-MDT0000 complete [12632.285595] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12635.233629] Key type lgssc unregistered [12635.540232] LNet: 431653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12635.552640] LNetError: 431653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12636.586395] LNet: Removed LNI 192.168.203.144@tcp [12637.543438] Key type .llcrypt unregistered [12637.546387] Key type ._llcrypt unregistered [12666.200368] Key type ._llcrypt registered [12666.203567] Key type .llcrypt registered [12666.297042] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [12683.369485] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [12684.572226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12684.740242] alg: No test for adler32 (adler32-zlib) [12685.913684] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [12686.300121] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [12688.033323] Key type lgssc registered [12689.197855] Lustre: Echo OBD driver; http://www.lustre.org/ [12731.553719] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [12751.766842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12751.797710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12753.067295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12753.097498] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12753.225489] Lustre: lustre-MDT0000: new disk, initializing [12753.344474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12753.367395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12759.029828] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12776.952516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12777.353075] Lustre: lustre-OST0000: new disk, initializing [12777.360491] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12777.364818] Lustre: Skipped 1 previous similar message [12777.371842] Lustre: 436689:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12777.478698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12784.208316] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12787.351217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12787.369593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12787.413344] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12794.175588] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12809.699438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12809.804440] Lustre: 435670:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12809.885433] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12810.006581] Lustre: lustre-MDT0001: new disk, initializing [12810.150356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12810.203850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12810.215159] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12814.958623] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12824.183664] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12824.350621] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12828.851276] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12829.051847] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12835.312529] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12835.315317] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12835.323444] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12835.333084] Lustre: Skipped 1 previous similar message [12835.358547] Lustre: Skipped 1 previous similar message [12837.495277] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.44@tcp (stopping) [12840.016292] Lustre: server umount lustre-MDT0001 complete [12845.536467] LustreError: 435682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12845.553937] LustreError: 435682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12850.660887] LustreError: 435681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12850.694139] LustreError: 435681:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12855.777996] LustreError: 435677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12855.799500] LustreError: 435677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12859.368374] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12859.383574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12859.410675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12860.907043] LustreError: 435682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12863.834574] Lustre: server umount lustre-OST0000 complete [12866.019135] LustreError: 435681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12882.399489] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12882.810805] Lustre: server umount lustre-MDT0000 complete [12897.944685] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12900.675958] Key type lgssc unregistered [12901.036686] LNet: 440443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12901.042446] LNetError: 440443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12901.054818] LNet: Removed LNI 192.168.203.144@tcp [12901.802737] Key type .llcrypt unregistered [12901.804528] Key type ._llcrypt unregistered [12942.271312] Key type ._llcrypt registered [12942.274488] Key type .llcrypt registered [12942.425281] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12942.770344] Key type .llcrypt unregistered [12942.773690] Key type ._llcrypt unregistered [12962.668218] Key type ._llcrypt registered [12962.682522] Key type .llcrypt registered [12962.795310] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [12963.508782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12963.577711] alg: No test for adler32 (adler32-zlib) [12964.702529] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [12965.028580] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [12966.695326] Key type lgssc registered [12968.012951] Lustre: Echo OBD driver; http://www.lustre.org/ [12995.193768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12995.232516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12996.487253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12996.533784] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12996.663337] Lustre: lustre-MDT0000: new disk, initializing [12996.766098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12996.794756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13002.451248] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13015.587496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13015.679896] Lustre: 443829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13015.750470] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13015.752959] Lustre: Skipped 1 previous similar message [13015.855748] Lustre: lustre-MDT0001: new disk, initializing [13016.018172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13016.069831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13016.088380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13021.449695] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13030.923376] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13037.019531] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13044.869412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13045.370344] Lustre: lustre-OST0000: new disk, initializing [13045.377791] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13045.394162] Lustre: 445795:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13045.541969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13052.227833] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13054.577906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13054.590645] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13054.712363] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13062.001694] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13070.308861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13070.314815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13070.326273] Lustre: Skipped 2 previous similar messages [13070.336861] Lustre: Skipped 2 previous similar messages [13074.631392] Lustre: server umount lustre-MDT0000 complete [13078.431597] LustreError: 444622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787128153 with bad export cookie 8082280270911551849 [13078.442484] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13078.447255] LustreError: 444622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13078.797914] Lustre: server umount lustre-MDT0001 complete [13092.915678] Lustre: server umount lustre-OST0000 complete [13103.953442] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 04:29:38 (1787128178) [13110.495270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13110.892196] LustreError: 447824:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13110.908377] LustreError: 447824:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13110.954675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13115.571805] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13116.401683] LustreError: 447825:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13120.484656] LustreError: 447824:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13122.762152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13123.073587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13126.955504] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13133.847272] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13140.053546] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13160.307324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13160.585910] Lustre: Found index 0 for lustre-OST0000, updating log [13160.782572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13167.264738] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13174.825664] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13181.237094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13191.652314] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13191.653512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13191.664310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13191.681466] Lustre: Skipped 1 previous similar message [13192.744140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13192.755183] Lustre: Skipped 2 previous similar messages [13196.983821] Lustre: server umount lustre-OST0000 complete [13214.181329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13214.187113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13214.208782] Lustre: Skipped 1 previous similar message [13218.338804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13218.345756] Lustre: Skipped 1 previous similar message [13218.969372] Lustre: server umount lustre-MDT0000 complete [13219.361195] LustreError: 447821:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13219.378486] LustreError: 447821:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13223.141916] LustreError: 447806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787128298 with bad export cookie 8082280270911553235 [13223.143939] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13223.155341] LustreError: 447806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13223.520278] Lustre: server umount lustre-MDT0001 complete [13228.194071] LNet: 1 local NIs in recovery (showing 1): 192.168.203.144@tcp [13230.783546] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 04:31:45 (1787128305) [13846.240333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13846.616283] LustreError: 454256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13846.685747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13851.168505] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13852.132437] LustreError: 454257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13856.226320] LustreError: 454256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13859.809238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13860.159352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13864.999431] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13872.528435] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13878.420423] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13898.471401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13898.781628] Lustre: Found index 0 for lustre-OST0000, updating log [13899.133731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13905.415460] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13907.103193] LNet: 1 local NIs in recovery (showing 1): 192.168.203.144@tcp [13913.362575] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13915.548686] Lustre: server umount lustre-OST0000 complete [13920.354849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13920.373160] Lustre: Skipped 1 previous similar message [13920.373634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13920.400617] Lustre: Skipped 1 previous similar message [13921.441679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13921.456746] Lustre: Skipped 1 previous similar message [13923.496339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13923.503965] Lustre: Skipped 3 previous similar messages [13925.719938] Lustre: server umount lustre-MDT0000 complete [13926.562171] LustreError: 454252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13926.591709] LustreError: 454252:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13929.772951] LustreError: 454237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787129004 with bad export cookie 8082280270911554327 [13929.782123] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13929.791898] LustreError: 454237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13930.240436] Lustre: server umount lustre-MDT0001 complete [13961.248913] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [13970.591422] LNet: 1 local NIs in recovery (showing 1): 192.168.203.144@tcp [13972.663753] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [14010.588301] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 04:44:44 (1787129084) [14030.865511] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [14033.775898] Key type lgssc unregistered [14034.094450] LNet: 460880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14034.099749] LNetError: 460880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14161.134785] LNet: Removed LNI 192.168.203.144@tcp [14161.943597] Key type .llcrypt unregistered [14161.947075] Key type ._llcrypt unregistered [14181.039214] Key type ._llcrypt registered [14181.043406] Key type .llcrypt registered [14181.123721] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [14182.310499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14182.440666] alg: No test for adler32 (adler32-zlib) [14183.569131] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [14183.812704] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [14185.495178] Key type lgssc registered [14186.697549] Lustre: Echo OBD driver; http://www.lustre.org/ [14191.364781] LNet: Added LNI 192.168.203.144@tcp100 [8/256/0/180] [14192.325791] LNet: Added LNI 192.168.203.144@tcp101 [8/256/0/180] [14193.303214] LNet: Added LNI 192.168.203.144@tcp102 [8/256/0/180] [14194.070135] LNet: Added LNI 192.168.203.144@tcp103 [8/256/0/180] [14194.996126] LNet: Added LNI 192.168.203.144@tcp104 [8/256/0/180] [14195.900440] LNet: Added LNI 192.168.203.144@tcp105 [8/256/0/180] [14196.899791] LNet: Added LNI 192.168.203.144@tcp106 [8/256/0/180] [14197.865485] LNet: Added LNI 192.168.203.144@tcp107 [8/256/0/180] [14198.470899] LNet: Added LNI 192.168.203.144@tcp108 [8/256/0/180] [14199.394599] LNet: Added LNI 192.168.203.144@tcp109 [8/256/0/180] [14200.516280] LNet: Added LNI 192.168.203.144@tcp110 [8/256/0/180] [14201.423755] LNet: Added LNI 192.168.203.144@tcp111 [8/256/0/180] [14202.391206] LNet: Added LNI 192.168.203.144@tcp112 [8/256/0/180] [14203.219993] LNet: Added LNI 192.168.203.144@tcp113 [8/256/0/180] [14204.277562] LNet: Added LNI 192.168.203.144@tcp114 [8/256/0/180] [14205.136751] LNet: Added LNI 192.168.203.144@tcp115 [8/256/0/180] [14206.092765] LNet: Added LNI 192.168.203.144@tcp116 [8/256/0/180] [14207.074296] LNet: Added LNI 192.168.203.144@tcp117 [8/256/0/180] [14208.219959] LNet: Added LNI 192.168.203.144@tcp118 [8/256/0/180] [14209.238082] LNet: Added LNI 192.168.203.144@tcp119 [8/256/0/180] [14210.075558] LNet: Added LNI 192.168.203.144@tcp120 [8/256/0/180] [14211.105647] LNet: Added LNI 192.168.203.144@tcp121 [8/256/0/180] [14212.634802] LNet: Added LNI 192.168.203.144@tcp122 [8/256/0/180] [14213.756896] LNet: Added LNI 192.168.203.144@tcp123 [8/256/0/180] [14214.991283] LNet: Added LNI 192.168.203.144@tcp124 [8/256/0/180] [14216.188861] LNet: Added LNI 192.168.203.144@tcp125 [8/256/0/180] [14217.236812] LNet: Added LNI 192.168.203.144@tcp126 [8/256/0/180] [14218.353406] LNet: Added LNI 192.168.203.144@tcp127 [8/256/0/180] [14219.336886] LNet: Added LNI 192.168.203.144@tcp128 [8/256/0/180] [14220.334882] LNet: Added LNI 192.168.203.144@tcp129 [8/256/0/180] [14221.251934] LNet: Added LNI 192.168.203.144@tcp130 [8/256/0/180] [14222.434832] LNet: Added LNI 192.168.203.144@tcp131 [8/256/0/180] [14223.381779] LNet: Added LNI 192.168.203.144@tcp132 [8/256/0/180] [14224.399555] LNet: Added LNI 192.168.203.144@tcp133 [8/256/0/180] [14225.327824] LNet: Added LNI 192.168.203.144@tcp134 [8/256/0/180] [14226.379951] LNet: Added LNI 192.168.203.144@tcp135 [8/256/0/180] [14233.566660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14233.619792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14234.779230] Lustre: 463916:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.144@tcp: can't add NID 192.168.203.144@tcp130: rc = -75 [14234.872411] LNetError: 461371:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.144 [14234.895468] LNetError: Protocol error connecting to 192.168.203.144@tcp116 at host 192.168.203.144:988: is it running a compatible version of Lustre? [14234.915728] LNetError: Skipped 7 previous similar messages [14234.983388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14235.007557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14235.039273] LNet: 6 local NIs in recovery (showing 6): 192.168.203.144@tcp125, 192.168.203.144@tcp126, 192.168.203.144@tcp127, 192.168.203.144@tcp128, 192.168.203.144@tcp129, 192.168.203.144@tcp116 [14235.069030] Lustre: lustre-MDT0000: new disk, initializing [14235.115887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14235.132237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14237.091071] LNetError: 463948:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.144 [14237.106377] LNetError: 463948:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 16 previous similar messages [14237.121510] LNetError: Protocol error connecting to 192.168.203.144@tcp125 at host 192.168.203.144:988: is it running a compatible version of Lustre? [14237.140684] LNetError: Skipped 9 previous similar messages [14239.918615] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [14241.185133] LNetError: 463943:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.144 [14241.192417] LNetError: 463943:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [14241.202087] LNetError: Protocol error connecting to 192.168.203.144@tcp125 at host 192.168.203.144:988: is it running a compatible version of Lustre? [14249.383737] LNetError: 463949:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.144 [14249.399462] LNetError: Protocol error connecting to 192.168.203.144@tcp125 at host 192.168.203.144:988: is it running a compatible version of Lustre? [14250.398099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14250.580070] Lustre: 463932:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14250.641948] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14250.646472] Lustre: Skipped 1 previous similar message [14250.718993] Lustre: lustre-MDT0001: new disk, initializing [14250.750659] Lustre: 464736:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.144@tcp: can't add NID 192.168.203.144@tcp130: rc = -75 [14250.765469] Lustre: 464736:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14250.903173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14250.960635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14250.976128] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14255.783541] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [14260.811464] Lustre: 465188:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.144@tcp: can't add NID 192.168.203.144@tcp130: rc = -75 [14260.825918] Lustre: 465188:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14264.737188] LNetError: 463941:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.144 [14264.751188] LNetError: Protocol error connecting to 192.168.203.144@tcp121 at host 192.168.203.144:988: is it running a compatible version of Lustre? [14265.393422] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14271.207507] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14272.932756] LNetError: 461373:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.144 [14272.955437] LNetError: 461373:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [14272.971555] LNetError: Protocol error connecting to 192.168.203.144@tcp120 at host 192.168.203.144:988: is it running a compatible version of Lustre? [14272.986336] LNetError: Skipped 4 previous similar messages [14278.736238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14279.099289] Lustre: lustre-OST0000: new disk, initializing [14279.102527] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14279.111176] Lustre: 465909:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14279.252240] Lustre: 465914:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.144@tcp: can't add NID 192.168.203.144@tcp130: rc = -75 [14279.261375] Lustre: 465914:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14279.329587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14286.392070] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [14288.938866] Lustre: 466367:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.144@tcp: can't add NID 192.168.203.144@tcp130: rc = -75 [14288.949693] Lustre: 466367:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14289.023657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14289.041518] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14289.168096] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14289.312498] LNetError: 463936:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.144 [14289.348290] LNetError: 463936:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [14289.367495] LNetError: Protocol error connecting to 192.168.203.144@tcp121 at host 192.168.203.144:988: is it running a compatible version of Lustre? [14289.386852] LNetError: Skipped 9 previous similar messages [14295.455153] LNet: 1 local NIs in recovery (showing 1): 192.168.203.144@tcp125 [14296.376770] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14304.737048] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14304.744596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14304.749474] LustreError: Skipped 1 previous similar message [14304.751498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14304.761820] Lustre: Skipped 1 previous similar message [14304.778096] Lustre: Skipped 1 previous similar message [14309.282043] Lustre: server umount lustre-OST0000 complete [14314.977021] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14314.988388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14319.601842] Lustre: server umount lustre-MDT0000 complete [14323.248229] LustreError: 464735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787129398 with bad export cookie 16589166390568967849 [14323.252133] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14323.263973] LustreError: 464735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14323.563069] Lustre: server umount lustre-MDT0001 complete [14328.224741] LNetError: 461372:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.144 [14328.236582] LNetError: 461372:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 14 previous similar messages [14328.242717] LNetError: Protocol error connecting to 192.168.203.144@tcp121 at host 192.168.203.144:988: is it running a compatible version of Lustre? [14328.249473] LNetError: Skipped 14 previous similar messages [14335.491861] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [14338.541625] Key type lgssc unregistered [14338.923520] LNet: 467825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14338.936752] LNetError: 467825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14339.999179] LNet: Removed LNI 192.168.203.144@tcp [14340.007582] LNetError: 467825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14340.027142] LNet: Removed LNI 192.168.203.144@tcp100 [14341.088164] LNet: Removed LNI 192.168.203.144@tcp101 [14341.093164] LNetError: 467825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14341.104046] LNetError: 467825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14341.124898] LNet: Removed LNI 192.168.203.144@tcp102 [14342.177788] LNet: Removed LNI 192.168.203.144@tcp103 [14343.199244] LNet: Removed LNI 192.168.203.144@tcp104 [14343.202287] LNetError: 467825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14343.220458] LNetError: 467825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14343.231934] LNet: Removed LNI 192.168.203.144@tcp105 [14344.287469] LNet: Removed LNI 192.168.203.144@tcp106 [14345.311164] LNet: Removed LNI 192.168.203.144@tcp107 [14346.335238] LNet: Removed LNI 192.168.203.144@tcp108 [14347.359158] LNet: Removed LNI 192.168.203.144@tcp109 [14347.373180] LNetError: 467825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14347.380468] LNetError: 467825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14347.389557] LNet: Removed LNI 192.168.203.144@tcp110 [14348.447183] LNet: Removed LNI 192.168.203.144@tcp111 [14349.472426] LNet: Removed LNI 192.168.203.144@tcp112 [14350.495171] LNet: Removed LNI 192.168.203.144@tcp113 [14351.519498] LNet: Removed LNI 192.168.203.144@tcp114 [14352.543160] LNet: Removed LNI 192.168.203.144@tcp115 [14353.567239] LNet: Removed LNI 192.168.203.144@tcp116 [14354.591309] LNet: Removed LNI 192.168.203.144@tcp117 [14355.615274] LNet: Removed LNI 192.168.203.144@tcp118 [14355.631234] LNetError: 467825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14355.645630] LNetError: 467825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14355.659525] LNet: Removed LNI 192.168.203.144@tcp119 [14356.703306] LNet: Removed LNI 192.168.203.144@tcp120 [14357.733934] LNet: Removed LNI 192.168.203.144@tcp121 [14358.815196] LNet: Removed LNI 192.168.203.144@tcp122 [14359.839258] LNet: Removed LNI 192.168.203.144@tcp123 [14360.863461] LNet: Removed LNI 192.168.203.144@tcp124 [14360.864980] LNet: Removed LNI 192.168.203.144@tcp125 [14361.887717] LNet: Removed LNI 192.168.203.144@tcp126 [14362.913264] LNet: Removed LNI 192.168.203.144@tcp127 [14363.937509] LNet: Removed LNI 192.168.203.144@tcp128 [14364.959325] LNet: Removed LNI 192.168.203.144@tcp129 [14364.963155] LNet: Removed LNI 192.168.203.144@tcp130 [14364.967784] LNet: Removed LNI 192.168.203.144@tcp131 [14364.974769] LNet: Removed LNI 192.168.203.144@tcp132 [14364.981420] LNet: Removed LNI 192.168.203.144@tcp133 [14364.983392] LNet: Removed LNI 192.168.203.144@tcp134 [14364.999850] LNet: Removed LNI 192.168.203.144@tcp135 [14366.188150] Key type .llcrypt unregistered [14366.196726] Key type ._llcrypt unregistered [14394.245578] Key type ._llcrypt registered [14394.249287] Key type .llcrypt registered [14394.441371] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [14412.164561] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [14413.520159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14413.593892] alg: No test for adler32 (adler32-zlib) [14414.725736] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [14415.101649] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [14416.784234] Key type lgssc registered [14417.924479] Lustre: Echo OBD driver; http://www.lustre.org/ [14455.873748] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 04:52:10 (1787129530) [14477.766211] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [14481.755564] Key type lgssc unregistered [14482.154450] LNet: 471917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14482.165780] LNetError: 471917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14482.180585] LNet: Removed LNI 192.168.203.144@tcp [14482.857572] Key type .llcrypt unregistered [14482.859199] Key type ._llcrypt unregistered [14500.780374] Key type ._llcrypt registered [14500.783946] Key type .llcrypt registered [14500.907774] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [14501.988607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14502.022158] alg: No test for adler32 (adler32-zlib) [14503.193139] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [14503.435929] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [14505.175238] Key type lgssc registered [14506.700792] Lustre: Echo OBD driver; http://www.lustre.org/ [14516.584535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14516.622043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14517.946388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14518.004542] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14518.084765] Lustre: lustre-MDT0000: new disk, initializing [14518.143087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14518.156181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14523.152210] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [14534.274387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14534.343773] Lustre: 473130:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14534.394966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14534.398842] Lustre: Skipped 1 previous similar message [14534.458326] Lustre: lustre-MDT0001: new disk, initializing [14534.546762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14534.585680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14534.591720] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14538.839243] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [14548.820863] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14556.318875] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14563.824896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14564.246349] Lustre: lustre-OST0000: new disk, initializing [14564.253102] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14564.274110] Lustre: 475100:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14564.363616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14571.090064] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [14574.187506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14574.195975] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14574.391456] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14581.475145] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14585.963824] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing lnet_if_list [14593.687783] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14593.737660] LNet: Added LNI 192.168.203.144@tcp10 [8/256/0/180] [14601.723450] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14601.757234] LNet: Added LNI 192.168.203.144@tcp11 [8/256/0/180] [14606.711638] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14606.736948] LNet: Added LNI 192.168.203.144@tcp12 [8/256/0/180] [14611.934422] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14611.990262] LNet: Added LNI 192.168.203.144@tcp13 [8/256/0/180] [14616.360636] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14616.407459] LNet: Added LNI 192.168.203.144@tcp14 [8/256/0/180] [14622.022664] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14622.059991] LNet: Added LNI 192.168.203.144@tcp15 [8/256/0/180] [14627.460577] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14627.497569] LNet: Added LNI 192.168.203.144@tcp16 [8/256/0/180] [14633.101386] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14633.130425] LNet: Added LNI 192.168.203.144@tcp17 [8/256/0/180] [14638.960572] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14639.018442] LNet: Added LNI 192.168.203.144@tcp18 [8/256/0/180] [14644.681165] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14644.724573] LNet: Added LNI 192.168.203.144@tcp19 [8/256/0/180] [14651.061864] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14651.102431] LNet: Added LNI 192.168.203.144@tcp20 [8/256/0/180] [14656.275447] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14656.293932] LNet: Added LNI 192.168.203.144@tcp21 [8/256/0/180] [14663.165173] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14663.204809] LNet: Added LNI 192.168.203.144@tcp22 [8/256/0/180] [14669.752692] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14669.828606] LNet: Added LNI 192.168.203.144@tcp23 [8/256/0/180] [14675.106695] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14675.144237] LNet: Added LNI 192.168.203.144@tcp24 [8/256/0/180] [14681.630780] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14681.664557] LNet: Added LNI 192.168.203.144@tcp25 [8/256/0/180] [14687.902585] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14687.960352] LNet: Added LNI 192.168.203.144@tcp26 [8/256/0/180] [14693.776233] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14693.799484] LNet: Added LNI 192.168.203.144@tcp27 [8/256/0/180] [14699.525922] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14699.575238] LNet: Added LNI 192.168.203.144@tcp28 [8/256/0/180] [14705.144639] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14705.172808] LNet: Added LNI 192.168.203.144@tcp29 [8/256/0/180] [14711.087272] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14711.132541] LNet: Added LNI 192.168.203.144@tcp30 [8/256/0/180] [14717.140813] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14717.161453] LNet: Added LNI 192.168.203.144@tcp31 [8/256/0/180] [14724.207203] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14724.231686] LNet: Added LNI 192.168.203.144@tcp32 [8/256/0/180] [14731.267296] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14731.296692] LNet: Added LNI 192.168.203.144@tcp33 [8/256/0/180] [14738.257971] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14738.302818] LNet: Added LNI 192.168.203.144@tcp34 [8/256/0/180] [14745.367358] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14745.400992] LNet: Added LNI 192.168.203.144@tcp35 [8/256/0/180] [14751.013872] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14751.060336] LNet: Added LNI 192.168.203.144@tcp36 [8/256/0/180] [14757.703672] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14757.732232] LNet: Added LNI 192.168.203.144@tcp37 [8/256/0/180] [14765.618683] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14765.694592] LNet: Added LNI 192.168.203.144@tcp38 [8/256/0/180] [14774.484912] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14774.544513] LNet: Added LNI 192.168.203.144@tcp39 [8/256/0/180] [14783.648450] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14783.689524] LNet: Added LNI 192.168.203.144@tcp40 [8/256/0/180] [14793.379147] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14793.424609] LNet: Added LNI 192.168.203.144@tcp41 [8/256/0/180] [14803.032380] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14803.080093] LNet: Added LNI 192.168.203.144@tcp42 [8/256/0/180] [14812.732509] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14812.778877] LNet: Added LNI 192.168.203.144@tcp43 [8/256/0/180] [14821.689982] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14821.718541] LNet: Added LNI 192.168.203.144@tcp44 [8/256/0/180] [14829.972809] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14830.040026] LNet: Added LNI 192.168.203.144@tcp45 [8/256/0/180] [14839.532732] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14839.571618] LNet: Added LNI 192.168.203.144@tcp46 [8/256/0/180] [14848.645829] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14848.682083] LNet: Added LNI 192.168.203.144@tcp47 [8/256/0/180] [14857.414198] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14857.455636] LNet: Added LNI 192.168.203.144@tcp48 [8/256/0/180] [14866.240960] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14866.278546] LNet: Added LNI 192.168.203.144@tcp49 [8/256/0/180] [14874.074625] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14874.121491] LNet: Added LNI 192.168.203.144@tcp50 [8/256/0/180] [14892.007048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14892.011511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14892.023078] Lustre: Skipped 2 previous similar messages [14892.045467] Lustre: Skipped 2 previous similar messages [14896.922067] Lustre: server umount lustre-MDT0000 complete [14900.384833] LustreError: 473124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787129975 with bad export cookie 2914608143144402920 [14900.392743] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14900.406957] LustreError: 473124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14900.727785] Lustre: server umount lustre-MDT0001 complete [14918.623121] Lustre: 472406:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787129977/real 1787129977] req@00000000ea23e941 x1873941184556032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787129993 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14918.660537] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14918.811987] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [14921.695191] Lustre: 472404:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787129980/real 1787129980] req@00000000cc616670 x1873941184556160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787129996 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14923.743501] Lustre: 472404:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787129982/real 1787129982] req@0000000092f1d92f x1873941184556288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787129998 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14926.706873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14926.815528] Lustre: 472405:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787129985/real 1787129985] req@00000000bc533be9 x1873941184556544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787130001 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14931.935133] Lustre: 472406:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787129990/real 1787129990] req@0000000038e8ea36 x1873941184556672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787130006 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14945.311842] LustreError: 472402:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d25d152c x1873941184557440/t0(0) o250->MGC192.168.203.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14945.605498] LustreError: 475094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14945.621099] LustreError: 475094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14945.655515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14947.682346] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14947.683293] LustreError: 475095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14949.733112] LustreError: 475094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14950.745488] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [14954.985083] LustreError: 475095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14958.861432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14959.243916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14963.512319] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [14964.395034] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14971.510422] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14977.387859] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15000.032179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15000.048317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15000.063853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15005.666289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15005.677417] Lustre: Skipped 2 previous similar messages [15010.794262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15010.801238] Lustre: Skipped 1 previous similar message [15013.343128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15013.611644] Lustre: server umount lustre-OST0000 complete [15020.514046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15020.531733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15020.548726] Lustre: Skipped 1 previous similar message [15020.560434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15023.311261] Lustre: server umount lustre-MDT0000 complete [15026.144558] LustreError: 499347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15026.161758] LustreError: 499347:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15026.675409] LustreError: 473123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130101 with bad export cookie 2914608143144405615 [15026.687627] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15026.701569] LustreError: 473123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15027.078870] Lustre: server umount lustre-MDT0001 complete [15038.959769] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [15041.594676] Key type lgssc unregistered [15041.997757] LNet: 502156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15042.007055] LNetError: 502156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15042.016773] LNet: Removed LNI 192.168.203.144@tcp [15042.021710] LNet: Removed LNI 192.168.203.144@tcp10 [15042.024699] LNet: Removed LNI 192.168.203.144@tcp11 [15042.027800] LNet: Removed LNI 192.168.203.144@tcp12 [15042.031910] LNet: Removed LNI 192.168.203.144@tcp13 [15042.035135] LNet: Removed LNI 192.168.203.144@tcp14 [15042.038737] LNet: Removed LNI 192.168.203.144@tcp15 [15042.045461] LNet: Removed LNI 192.168.203.144@tcp16 [15042.048984] LNet: Removed LNI 192.168.203.144@tcp17 [15042.054954] LNet: Removed LNI 192.168.203.144@tcp18 [15042.062333] LNet: Removed LNI 192.168.203.144@tcp19 [15042.070685] LNet: Removed LNI 192.168.203.144@tcp20 [15042.074217] LNet: Removed LNI 192.168.203.144@tcp21 [15042.076664] LNet: Removed LNI 192.168.203.144@tcp22 [15042.078514] LNet: Removed LNI 192.168.203.144@tcp23 [15042.082400] LNet: Removed LNI 192.168.203.144@tcp24 [15042.085679] LNet: Removed LNI 192.168.203.144@tcp25 [15042.087988] LNet: Removed LNI 192.168.203.144@tcp26 [15042.089909] LNet: Removed LNI 192.168.203.144@tcp27 [15042.093475] LNet: Removed LNI 192.168.203.144@tcp28 [15042.096193] LNet: Removed LNI 192.168.203.144@tcp29 [15042.098734] LNet: Removed LNI 192.168.203.144@tcp30 [15042.104532] LNet: Removed LNI 192.168.203.144@tcp31 [15042.107149] LNet: Removed LNI 192.168.203.144@tcp32 [15042.109749] LNet: Removed LNI 192.168.203.144@tcp33 [15042.112071] LNet: Removed LNI 192.168.203.144@tcp34 [15042.113521] LNet: Removed LNI 192.168.203.144@tcp35 [15042.115654] LNet: Removed LNI 192.168.203.144@tcp36 [15042.118257] LNet: Removed LNI 192.168.203.144@tcp37 [15042.120454] LNet: Removed LNI 192.168.203.144@tcp38 [15042.122782] LNet: Removed LNI 192.168.203.144@tcp39 [15042.124899] LNet: Removed LNI 192.168.203.144@tcp40 [15042.127909] LNet: Removed LNI 192.168.203.144@tcp41 [15042.130736] LNet: Removed LNI 192.168.203.144@tcp42 [15042.133278] LNet: Removed LNI 192.168.203.144@tcp43 [15042.135860] LNet: Removed LNI 192.168.203.144@tcp44 [15042.138622] LNet: Removed LNI 192.168.203.144@tcp45 [15042.141346] LNet: Removed LNI 192.168.203.144@tcp46 [15042.143870] LNet: Removed LNI 192.168.203.144@tcp47 [15042.146667] LNet: Removed LNI 192.168.203.144@tcp48 [15042.149535] LNet: Removed LNI 192.168.203.144@tcp49 [15042.164566] LNet: Removed LNI 192.168.203.144@tcp50 [15042.822423] Key type .llcrypt unregistered [15042.825510] Key type ._llcrypt unregistered [15062.851898] Key type ._llcrypt registered [15062.857958] Key type .llcrypt registered [15062.994525] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [15076.901504] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [15078.050157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15078.068186] alg: No test for adler32 (adler32-zlib) [15079.197421] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [15079.452349] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [15081.159505] Key type lgssc registered [15082.348973] Lustre: Echo OBD driver; http://www.lustre.org/ [15117.162376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15117.199973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15118.494650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15118.561552] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15118.704469] Lustre: lustre-MDT0000: new disk, initializing [15118.842340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15118.860436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15123.548393] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [15133.987995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15134.199973] Lustre: 505353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15134.288288] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15134.292438] Lustre: Skipped 1 previous similar message [15134.347975] Lustre: lustre-MDT0001: new disk, initializing [15134.429468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15134.466534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15134.481906] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15139.506479] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [15148.294535] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15153.146537] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15159.768097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15159.975194] Lustre: *** cfs_fail_loc=90f, val=0*** [15160.103189] Lustre: lustre-OST0000: new disk, initializing [15160.107525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15160.117517] Lustre: 507367:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15160.204812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15165.991443] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [15167.021907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15167.038997] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15167.159562] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15174.864784] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15182.819677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15182.835397] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15182.839954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15182.849414] Lustre: Skipped 1 previous similar message [15187.577620] Lustre: server umount lustre-OST0000 complete [15193.057237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15193.089455] Lustre: Skipped 1 previous similar message [15193.096627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15193.108195] Lustre: Skipped 1 previous similar message [15198.172787] Lustre: server umount lustre-MDT0000 complete [15202.320435] LustreError: 505345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130277 with bad export cookie 14635214012719086162 [15202.321100] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15202.343506] LustreError: 505345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15202.778572] Lustre: server umount lustre-MDT0001 complete [15213.377790] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [15216.463201] Key type lgssc unregistered [15216.871800] LNet: 509271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15216.881728] LNetError: 509271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15217.974522] LNet: Removed LNI 192.168.203.144@tcp [15218.929599] Key type .llcrypt unregistered [15218.933122] Key type ._llcrypt unregistered [15239.060144] Key type ._llcrypt registered [15239.065621] Key type .llcrypt registered [15239.179417] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [15240.383909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15240.408349] alg: No test for adler32 (adler32-zlib) [15241.488837] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [15241.770269] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [15243.433045] Key type lgssc registered [15244.539623] Lustre: Echo OBD driver; http://www.lustre.org/ [15251.230497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15251.269609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15252.662058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15256.584463] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [15262.178064] LustreError: 510545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15262.200530] LustreError: 510545:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15264.438698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15264.921601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15269.505512] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [15275.939893] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15281.361882] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15287.032080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15287.322891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15293.553218] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [15302.604195] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15313.887659] loop0: detected capacity change from 0 to 204800000 [15314.081183] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15314.101889] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15314.118904] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15320.044846] loop0: detected capacity change from 0 to 204800000 [15320.195108] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15320.206324] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15320.217955] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15325.708513] loop0: detected capacity change from 0 to 204800000 [15331.048285] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15331.240505] Lustre: fs15246-OST0000: new disk, initializing [15331.244687] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15331.253342] Lustre: 514137:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15331.351377] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15337.890343] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [15348.039177] loop1: detected capacity change from 0 to 204800000 [15353.131979] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15353.218981] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15353.265728] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15353.322307] Lustre: fs15246-MDT0000: new disk, initializing [15353.440965] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15353.466835] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15353.577609] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15354.599222] Lustre: 515259:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000004ea86003 x1873941958276736/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787130445 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15357.504422] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [15363.746380] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15363.768164] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15363.823305] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15381.480666] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15381.506804] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15385.899863] Lustre: server umount fs15246-MDT0000 complete [15386.601924] LustreError: 510541:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15391.716186] LustreError: 510541:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15395.120835] Lustre: server umount fs15246-OST0000 complete [15410.147218] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15410.166983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15410.188164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15412.200622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15412.205669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15412.211130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15412.234066] Lustre: Skipped 1 previous similar message [15413.670814] Lustre: server umount lustre-OST0000 complete [15418.337884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15418.346443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15418.365282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15422.433327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15422.438440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15423.724193] Lustre: server umount lustre-MDT0000 complete [15427.485864] LustreError: 510524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130502 with bad export cookie 38252372704994802 [15427.508495] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15427.509892] LustreError: 510524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15427.740706] Lustre: server umount lustre-MDT0001 complete [15437.728754] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [15440.246280] Key type lgssc unregistered [15440.493323] LNet: 518355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15440.504797] LNetError: 518355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15440.531520] LNet: Removed LNI 192.168.203.144@tcp [15441.293189] Key type .llcrypt unregistered [15441.294780] Key type ._llcrypt unregistered [15460.822959] Key type ._llcrypt registered [15460.825485] Key type .llcrypt registered [15460.982693] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [15461.912944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15462.054625] alg: No test for adler32 (adler32-zlib) [15463.138514] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [15463.465798] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [15465.175280] Key type lgssc registered [15466.383245] Lustre: Echo OBD driver; http://www.lustre.org/ [15507.118852] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [15519.938759] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [15558.234778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15558.273839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15559.566478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15559.593715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15559.644167] Lustre: lustre-MDT0000: new disk, initializing [15559.692670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15559.706203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15564.532576] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [15578.144386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15578.343299] Lustre: 522822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15578.460131] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15578.471107] Lustre: Skipped 1 previous similar message [15578.641613] Lustre: lustre-MDT0001: new disk, initializing [15578.775196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15578.889702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15578.910651] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15584.107779] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [15593.447821] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15599.942440] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15607.352543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15607.558895] Lustre: lustre-OST0000: new disk, initializing [15607.561232] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15607.567226] Lustre: 524795:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15607.645592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15613.477237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15613.481572] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15613.592221] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15614.303215] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [15623.203744] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15628.316284] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15628.615408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15633.167391] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15633.397562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15639.015610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15639.036216] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15639.051847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15639.056924] Lustre: Skipped 1 previous similar message [15641.440347] Lustre: server umount lustre-OST0000 complete [15645.666099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15645.671366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15645.696404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15645.709030] Lustre: Skipped 1 previous similar message [15649.250000] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15649.250805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15649.272923] Lustre: Skipped 1 previous similar message [15651.332757] Lustre: server umount lustre-MDT0000 complete [15655.001317] LustreError: 524794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130730 with bad export cookie 7714177979160949539 [15655.005474] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15655.009494] LustreError: 524794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15655.270226] Lustre: server umount lustre-MDT0001 complete [15677.673968] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [15688.637333] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [15719.250512] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15460 sec ========= 05:13:13 (1787130793) [15721.145494] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:13:15 (1787130795) === [15725.387979] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:13:19 (1787130799) === [15762.046526] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [15764.466687] Key type lgssc unregistered [15764.701955] LNet: 531434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15764.710584] LNetError: 531434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15764.734388] LNet: Removed LNI 192.168.203.144@tcp [15765.482984] Key type .llcrypt unregistered [15765.488594] Key type ._llcrypt unregistered