[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 503395111 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002329] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.006870] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008009] pid_max: default: 32768 minimum: 301 [ 0.010130] LSM: Security Framework initializing [ 0.011038] Yama: becoming mindful. [ 0.012027] SELinux: Initializing. [ 0.013056] *** VALIDATE selinux *** [ 0.021249] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027218] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028172] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029102] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030172] *** VALIDATE tmpfs *** [ 0.032210] *** VALIDATE proc *** [ 0.033316] *** VALIDATE cgroup *** [ 0.034015] *** VALIDATE cgroup2 *** [ 0.035306] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036218] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038038] Spectre V2 : User space: Vulnerable [ 0.039000] Speculative Store Bypass: Vulnerable [ 0.045037] debug: unmapping init [mem 0xffffffffb6059000-0xffffffffb6060fff] [ 0.048662] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051068] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052030] ... version: 2 [ 0.053018] ... bit width: 48 [ 0.054021] ... generic registers: 4 [ 0.055017] ... value mask: 0000ffffffffffff [ 0.056022] ... max period: 00007fffffffffff [ 0.058011] ... fixed-purpose events: 3 [ 0.059007] ... event mask: 000000070000000f [ 0.060366] rcu: Hierarchical SRCU implementation. [ 0.063812] smp: Bringing up secondary CPUs ... [ 0.064817] x86: Booting SMP configuration: [ 0.065046] .... node #0, CPUs: #1 #2 #3 [ 0.072493] smp: Brought up 1 node, 4 CPUs [ 0.074011] smpboot: Max logical packages: 1 [ 0.074975] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.150019] node 0 deferred pages initialised in 74ms [ 0.152151] devtmpfs: initialized [ 0.153057] x86/mm: Memory block size: 128MB [ 0.155334] gcov: version magic: 0x41383552 [ 0.157516] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.158070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.159237] pinctrl core: initialized pinctrl subsystem [ 0.160168] [ 0.160606] ************************************************************* [ 0.161008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162023] ** ** [ 0.163008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164007] ** ** [ 0.165010] ** This means that this kernel is built to expose internal ** [ 0.166008] ** IOMMU data structures, which may compromise security on ** [ 0.167009] ** your system. ** [ 0.168007] ** ** [ 0.169007] ** If you see this message and you are not debugging the ** [ 0.170008] ** kernel, report this immediately to your vendor! ** [ 0.171011] ** ** [ 0.172013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173014] ************************************************************* [ 0.175041] NET: Registered protocol family 16 [ 0.176824] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.177063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.178050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.180007] cpuidle: using governor menu [ 0.181000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.182312] PCI: Using configuration type 1 for base access [ 0.184135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.193148] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196046] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199249] cryptd: max_cpu_qlen set to 1000 [ 0.201350] ACPI: Added _OSI(Module Device) [ 0.202010] ACPI: Added _OSI(Processor Device) [ 0.203008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.204009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.208713] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.213492] ACPI: Interpreter enabled [ 0.214050] ACPI: PM: (supports S0 S3 S4 S5) [ 0.215012] ACPI: Using IOAPIC for interrupt routing [ 0.216085] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.217428] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.230433] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.231033] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.232015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.233086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.235681] acpiphp: Slot [2] registered [ 0.236228] acpiphp: Slot [5] registered [ 0.237111] acpiphp: Slot [6] registered [ 0.238049] acpiphp: Slot [7] registered [ 0.239087] acpiphp: Slot [8] registered [ 0.241110] acpiphp: Slot [9] registered [ 0.242166] acpiphp: Slot [10] registered [ 0.243118] acpiphp: Slot [3] registered [ 0.244101] acpiphp: Slot [4] registered [ 0.245077] acpiphp: Slot [11] registered [ 0.246128] acpiphp: Slot [12] registered [ 0.247049] acpiphp: Slot [13] registered [ 0.247955] acpiphp: Slot [14] registered [ 0.248050] acpiphp: Slot [15] registered [ 0.249077] acpiphp: Slot [16] registered [ 0.250079] acpiphp: Slot [17] registered [ 0.251136] acpiphp: Slot [18] registered [ 0.252061] acpiphp: Slot [19] registered [ 0.253080] acpiphp: Slot [20] registered [ 0.254135] acpiphp: Slot [21] registered [ 0.255077] acpiphp: Slot [22] registered [ 0.256071] acpiphp: Slot [23] registered [ 0.257077] acpiphp: Slot [24] registered [ 0.259105] acpiphp: Slot [25] registered [ 0.261135] acpiphp: Slot [26] registered [ 0.263098] acpiphp: Slot [27] registered [ 0.264062] acpiphp: Slot [28] registered [ 0.265000] acpiphp: Slot [29] registered [ 0.265000] acpiphp: Slot [30] registered [ 0.266065] acpiphp: Slot [31] registered [ 0.271031] PCI host bridge to bus 0000:00 [ 0.272030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.273025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.311037] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.313022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.315015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.317090] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.319176] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.321270] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.325661] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.334000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.334000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.334000] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.334000] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.334000] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.334000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.334000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.334000] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.334000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.334000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.334000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.334000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.334000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.334000] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.341024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.365020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.372000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.374000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.374000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.391018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.392000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.392000] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.392000] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.404015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.414000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.414000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.414000] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.424025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.441498] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.452014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.466022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.485015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.502804] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.513022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.522022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.546019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.563475] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.565479] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.568444] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.570358] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.572193] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.578133] iommu: Default domain type: Passthrough [ 0.580390] SCSI subsystem initialized [ 0.582111] ACPI: bus type USB registered [ 0.583156] usbcore: registered new interface driver usbfs [ 0.585062] usbcore: registered new interface driver hub [ 0.586063] usbcore: registered new device driver usb [ 0.588175] pps_core: LinuxPPS API ver. 1 registered [ 0.590008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.592046] PTP clock support registered [ 0.616053] EDAC MC: Ver: 3.0.0 [ 0.617142] PCI: Using ACPI for IRQ routing [ 0.618000] NetLabel: Initializing [ 0.618000] NetLabel: domain hash size = 128 [ 0.620009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.621141] NetLabel: unlabeled traffic allowed by default [ 0.625109] vgaarb: loaded [ 0.627038] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.628010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.634685] clocksource: Switched to clocksource kvm-clock [ 0.800320] VFS: Disk quotas dquot_6.6.0 [ 0.802122] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.806189] *** VALIDATE ramfs *** [ 0.808492] *** VALIDATE hugetlbfs *** [ 0.812382] pnp: PnP ACPI init [ 0.816602] pnp: PnP ACPI: found 6 devices [ 0.836819] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.839849] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.841476] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.843542] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.845832] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.847660] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.849895] NET: Registered protocol family 2 [ 0.852206] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.857000] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.860254] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.864503] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.867687] TCP: Hash tables configured (established 65536 bind 65536) [ 0.870461] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.873580] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.876251] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.878958] NET: Registered protocol family 1 [ 0.881393] RPC: Registered named UNIX socket transport module. [ 0.883679] RPC: Registered udp transport module. [ 0.885306] RPC: Registered tcp transport module. [ 0.887215] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.889802] NET: Registered protocol family 44 [ 0.891446] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.895946] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.898285] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.902665] PCI: CLS 0 bytes, default 64 [ 0.904853] Unpacking initramfs... [ 3.097401] debug: unmapping init [mem 0xffff9e437cc54000-0xffff9e437ffbffff] [ 3.104684] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.107181] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.111693] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.718276] Initialise system trusted keyrings [ 3.720392] Key type blacklist registered [ 3.976695] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.004458] zbud: loaded [ 4.016054] *** VALIDATE nfs *** [ 4.016863] *** VALIDATE nfs4 *** [ 4.019371] pstore: using deflate compression [ 4.024249] Platform Keyring initialized [ 4.135485] NET: Registered protocol family 38 [ 4.136905] Key type asymmetric registered [ 4.137923] Asymmetric key parser 'x509' registered [ 4.139356] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.141355] io scheduler mq-deadline registered [ 4.142808] io scheduler kyber registered [ 4.144297] io scheduler bfq registered [ 4.145788] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.147866] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.149827] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.151910] ACPI: Power Button [PWRF] [ 4.157468] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.163244] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.176901] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.182405] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.194534] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.219448] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.244253] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.247847] Non-volatile memory driver v1.3 [ 4.248933] Linux agpgart interface v0.103 [ 4.274772] virtio_blk virtio1: [vda] 146536 512-byte logical blocks (75.0 MB/71.6 MiB) [ 4.279251] vda: detected capacity change from 0 to 75026432 [ 4.301261] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.304810] vdb: detected capacity change from 0 to 1073741824 [ 4.329355] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.334914] vdc: detected capacity change from 0 to 2621440000 [ 4.360454] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.363560] vdd: detected capacity change from 0 to 2621440000 [ 4.377795] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.380448] vde: detected capacity change from 0 to 4294967296 [ 4.395442] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.398178] vdf: detected capacity change from 0 to 4294967296 [ 4.405225] libphy: Fixed MDIO Bus: probed [ 4.412194] usbcore: registered new interface driver usbserial_generic [ 4.414408] usbserial: USB Serial support registered for generic [ 4.417193] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.420934] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.422520] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.424751] mousedev: PS/2 mouse device common for all mice [ 4.428452] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.432541] rtc_cmos 00:05: RTC can wake from S4 [ 4.438018] rtc_cmos 00:05: registered as rtc0 [ 4.438885] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.439511] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.445884] intel_pstate: CPU model not supported [ 4.448229] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.450152] hid: raw HID events driver (C) Jiri Kosina [ 4.455054] usbcore: registered new interface driver usbhid [ 4.456699] usbhid: USB HID core driver [ 4.458172] drop_monitor: Initializing network drop monitor service [ 4.460325] Initializing XFRM netlink socket [ 4.462055] NET: Registered protocol family 10 [ 4.464731] Segment Routing with IPv6 [ 4.466282] NET: Registered protocol family 17 [ 4.468110] mpls_gso: MPLS GSO support [ 4.474269] RAS: Correctable Errors collector initialized. [ 4.476121] AVX version of gcm_enc/dec engaged. [ 4.477435] AES CTR mode by8 optimization enabled [ 4.635885] sched_clock: Marking stable (4635835972, 0)->(6788335649, -2152499677) [ 4.642344] registered taskstats version 1 [ 4.645531] Loading compiled-in X.509 certificates [ 4.650392] zswap: loaded using pool lzo/zbud [ 4.698884] Key type big_key registered [ 4.719670] Key type encrypted registered [ 4.724350] ima: No TPM chip found, activating TPM-bypass! [ 4.726812] ima: Allocated hash algorithm: sha1 [ 4.728785] ima: No architecture policies found [ 4.730455] evm: Initialising EVM extended attributes: [ 4.735271] evm: security.selinux [ 4.736315] evm: security.ima [ 4.737372] evm: security.capability [ 4.738550] evm: HMAC attrs: 0x1 [ 4.742364] rtc_cmos 00:05: setting system clock to 2026-08-22 04:27:40 UTC (1787372860) [ 4.754288] debug: unmapping init [mem 0xffffffffb7003000-0xffffffffb71fffff] [ 4.757213] debug: unmapping init [mem 0xffffffffb5d82000-0xffffffffb6058fff] [ 4.771211] Write protecting the kernel read-only data: 28672k [ 4.774229] debug: unmapping init [mem 0xffffffffb4403000-0xffffffffb45fffff] [ 4.776850] debug: unmapping init [mem 0xffffffffb4d14000-0xffffffffb4dfffff] [ 4.842765] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.855494] systemd[1]: Detected virtualization kvm. [ 4.859551] systemd[1]: Detected architecture x86-64. [ 4.861606] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.892279] systemd[1]: No hostname configured. [ 4.896874] systemd[1]: Set hostname to . [ 4.898578] random: systemd: uninitialized urandom read (16 bytes read) [ 4.900810] systemd[1]: Initializing machine ID from random generator. [ 5.018039] random: ln: uninitialized urandom read (6 bytes read) [ 5.315435] random: systemd: uninitialized urandom read (16 bytes read) [ 5.317692] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 5.321962] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.326769] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.728839] device-mapper: uevent: version 1.0.3 [ 6.731524] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook...[ 8.209789] random: fast init done [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.358964] virtio_net virtio0 ens2: renamed from eth0 [ 8.656109] scsi host0: ata_piix [ 8.740885] scsi host1: ata_piix [ 8.744386] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.752844] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.249267] random: crng init done [ 14.250852] random: 7 urandom warning(s) missed due to ratelimiting [ 16.697987] dracut-initqueue[594]: RTNETLINK answers: File exists Starting nbd nbd0... [ 17.915121] hrtimer: interrupt took 6269104 ns [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK [ 18.608764] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.240843] printk: systemd: 22 output lines suppressed due to ratelimiting [ 24.706403] SELinux: Disabled at runtime. [ 24.828889] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.837763] systemd[1]: Detected virtualization kvm. [ 24.839429] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 30.865385] systemd[1]: initrd-switch-root.service: Succeeded. [ 30.869607] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 30.881741] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 30.888604] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 30.892244] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 30.915029] systemd[1]: Starting Journal Service... Starting Journal Service... [ 31.000639] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 31.006181] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ 31.015735] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ 31.106721] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ 31.767116] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 35.664517] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 39.620850] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 39.967557] input: PC Speaker as /devices/platform/pcspkr/input/input5 [* ] A start job is running for Confi[ 42.192457] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer gur…only root support (11s / no limit) [** ] A start job is running for Configur…only root support (12s / no limit)[ 42.794889] EDAC sbridge: Ver: 1.1.2 [*** ] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [ ***] A start job is running for Configur…only root support (15s / no limit) [ **] A start job is running for Configur…only root support (16s / no limit) [ *] A start job is running for Configur…only root support (17s / no limit) [ **] A start job is running for Configur…only root support (18s / no limit) [ ***] A start job is running for Configur…only root support (18s / no limit) [ *** ] A start job is running for Configur…only root support (19s / no limit)[ 49.482368] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (19s / no limit)[ 49.961289] NFS: Registering the id_resolver key type [ 49.962706] Key type id_resolver registered [ 49.963936] Key type id_legacy registered [*** ] A start job is running for Configur…only root support (20s / no limit) [** ] A start job is running for Configur…only root support (20s / no limit) [* ] A start job is running for Configur…only root support (21s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg218-server login: [ 152.253113] libcfs: loading out-of-tree module taints kernel. [ 152.384452] Key type ._llcrypt registered [ 152.386938] Key type .llcrypt registered [ 152.538768] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 170.325618] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 171.730930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 171.759265] alg: No test for adler32 (adler32-zlib) [ 173.180508] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 174.095390] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 175.832395] Key type lgssc registered [ 177.362606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 191.688646] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 234.141642] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 248.395877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 248.440498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 249.773572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 249.851737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 249.998813] Lustre: lustre-MDT0000: new disk, initializing [ 250.108396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 250.143055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 254.798983] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 269.289175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 269.408381] Lustre: 6536:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 269.442176] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 269.448466] Lustre: Skipped 1 previous similar message [ 269.572266] Lustre: lustre-MDT0001: new disk, initializing [ 269.692707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 269.714188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 269.727984] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 274.239803] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 279.705411] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 289.021282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 289.329577] Lustre: lustre-OST0000: new disk, initializing [ 289.337117] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 289.353059] Lustre: 8475:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 289.433018] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 294.444456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 294.452814] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 294.551498] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 295.916313] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 309.896625] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 310.028663] Lustre: lustre-OST0001: new disk, initializing [ 310.033526] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 310.039346] Lustre: 9546:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 310.096058] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 316.461230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 316.473268] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 316.545316] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 317.660381] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 332.689975] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 343.710045] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 351.945048] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing check_logdir /tmp/testlogs/ [ 357.991953] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing yml_node [ 363.123419] Lustre: DEBUG MARKER: Client: 2.17.57.46 [ 366.195715] Lustre: DEBUG MARKER: MDS: 2.17.57.46 [ 368.942460] Lustre: DEBUG MARKER: OSS: 2.17.57.46 [ 370.667991] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 00:33:44 EDT 2026 [ 391.753676] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 394.247786] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 413.156566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 413.170631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 413.193523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 417.751156] Lustre: server umount lustre-MDT0000 complete [ 421.879429] LustreError: 6528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787373277 with bad export cookie 12935663690654981457 [ 421.881757] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 421.887654] LustreError: 6528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 423.906126] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 423.909522] LustreError: 12041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 423.909928] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 423.909934] Lustre: Skipped 3 previous similar messages [ 423.937790] Lustre: Skipped 4 previous similar messages [ 423.959113] LustreError: 12041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 426.914895] LustreError: 6543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 426.915188] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 426.929632] LustreError: 6543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 426.936425] Lustre: Skipped 2 previous similar messages [ 428.341714] Lustre: server umount lustre-MDT0001 complete [ 433.131381] Lustre: server umount lustre-OST0000 complete [ 438.649157] Lustre: server umount lustre-OST0001 complete [ 449.616337] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 461.329971] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 501.867774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 502.238552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 502.300643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 502.494228] Lustre: lustre-MDT0000: new disk, initializing [ 502.606360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 502.633772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 508.314793] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 519.919937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 520.088732] Lustre: 15107:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 520.153057] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 520.158395] Lustre: Skipped 1 previous similar message [ 520.259926] Lustre: lustre-MDT0001: new disk, initializing [ 520.407461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 520.451922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 520.464693] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 525.307725] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 535.607720] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 542.341437] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 551.014301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 551.393611] Lustre: lustre-OST0000: new disk, initializing [ 551.401787] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 551.413680] Lustre: 17075:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 551.522046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 553.294473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 553.301847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 553.391589] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 558.003498] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 568.660159] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 572.405311] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 572.603646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 576.819700] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 577.050187] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 579.046581] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 579.050698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.058961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 579.061711] Lustre: Skipped 1 previous similar message [ 584.170144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 584.176130] Lustre: Skipped 2 previous similar messages [ 592.864468] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 593.035648] Lustre: server umount lustre-OST0000 complete [ 597.479414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 597.482088] LustreError: Skipped 1 previous similar message [ 597.489496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 597.518523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 597.524979] Lustre: Skipped 2 previous similar messages [ 604.641885] LustreError: 15114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 604.657297] LustreError: 15114:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 604.745434] Lustre: server umount lustre-MDT0000 complete [ 608.501694] LustreError: 15100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787373464 with bad export cookie 12935663690654982990 [ 608.517856] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 608.520683] LustreError: 15100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 608.827566] Lustre: server umount lustre-MDT0001 complete [ 613.068869] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 00:37:46 (1787373466) [ 615.528300] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 617.005255] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 00:37:51 (1787373471) [ 623.703916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 624.204571] LustreError: 19487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 624.264700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 628.851984] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 629.729820] LustreError: 19488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 637.803045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 642.756237] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 651.298174] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 657.625347] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 665.051842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 672.676343] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 679.907664] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 693.221115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 693.240632] Lustre: Skipped 1 previous similar message [ 693.262268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 693.265883] Lustre: Skipped 3 previous similar messages [ 697.452897] Lustre: server umount lustre-OST0000 complete [ 703.472050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 703.487670] Lustre: Skipped 1 previous similar message [ 707.727289] Lustre: server umount lustre-MDT0000 complete [ 708.581099] LustreError: 19487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 708.618671] LustreError: 19487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 711.310375] LustreError: 19469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787373567 with bad export cookie 12935663690654983865 [ 711.315831] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 711.322233] LustreError: 19469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 711.922474] Lustre: server umount lustre-MDT0001 complete [ 724.631951] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 728.043975] Key type lgssc unregistered [ 728.394809] LNet: 23025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 728.399280] LNetError: 23025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 729.448534] LNet: Removed LNI 192.168.202.118@tcp [ 730.625163] Key type .llcrypt unregistered [ 730.627294] Key type ._llcrypt unregistered [ 757.104809] Key type ._llcrypt registered [ 757.107635] Key type .llcrypt registered [ 757.237489] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 758.379503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 758.433891] alg: No test for adler32 (adler32-zlib) [ 759.689567] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 760.028631] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 761.728207] Key type lgssc registered [ 763.025511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 772.944261] loop: module loaded [ 773.047290] loop0: detected capacity change from 0 to 204800000 [ 773.330963] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 773.337958] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 773.770813] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 779.533400] loop0: detected capacity change from 0 to 204800000 [ 779.746281] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 779.762683] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 779.784233] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 784.905954] loop0: detected capacity change from 0 to 204800000 [ 790.462653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 790.509702] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 791.901945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 791.961330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 792.100730] Lustre: lustre-MDT0000: new disk, initializing [ 792.230539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 792.280576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 797.518688] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 806.383157] loop1: detected capacity change from 0 to 204800000 [ 812.152168] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 812.563118] Lustre: lustre-OST07c6: new disk, initializing [ 812.566391] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 812.573074] Lustre: Skipped 1 previous similar message [ 812.578493] Lustre: 26216:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 812.688839] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 818.250304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 818.258970] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 818.298130] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 819.124993] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 825.454681] Lustre: 25205:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 825.461049] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 825.466620] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 825.470708] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 825.475319] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 825.479437] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 843.420592] Lustre: 26731:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 843.429622] Lustre: 26731:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 843.436768] Lustre: 26731:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 843.447601] Lustre: 26731:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 843.474129] Lustre: 26731:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 843.484626] Lustre: 26731:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 848.603933] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 848.609651] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 848.623257] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 848.638711] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 848.649362] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 848.655882] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 848.666153] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 848.673857] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 848.677515] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 848.682930] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 848.688619] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 848.694422] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 851.911426] Lustre: 26731:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 851.921854] Lustre: 26731:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 851.935690] Lustre: 26731:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 851.943611] Lustre: 26731:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 851.950332] Lustre: 26731:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 851.957924] Lustre: 26731:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 851.968955] Lustre: 26731:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 851.971248] Lustre: 26731:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 851.977200] Lustre: 26731:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 851.981532] Lustre: 26731:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 851.986815] Lustre: 26731:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 851.992059] Lustre: 26731:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 856.875062] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 856.882941] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 856.889909] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 856.896885] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 856.904951] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 856.922787] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 856.930119] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 856.943679] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 856.949697] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 856.955546] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 856.963145] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 856.970126] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 865.654759] Lustre: 26731:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 865.661351] Lustre: 26731:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 865.664620] Lustre: 26731:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 865.668453] Lustre: 26731:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 865.672240] Lustre: 26731:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 865.676361] Lustre: 26731:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 865.679501] Lustre: 26731:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 865.683731] Lustre: 26731:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 865.687489] Lustre: 26731:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 865.695766] Lustre: 26731:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 865.704880] Lustre: 26731:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 865.720348] Lustre: 26731:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 920.545258] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 920.565971] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 925.989616] Lustre: server umount lustre-OST07c6 complete [ 935.849467] Lustre: server umount lustre-MDT0000 complete [ 953.009244] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 955.702378] Key type lgssc unregistered [ 956.008040] LNet: 28782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.016277] LNetError: 28782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.030062] LNet: Removed LNI 192.168.202.118@tcp [ 956.831884] Key type .llcrypt unregistered [ 956.835106] Key type ._llcrypt unregistered [ 975.555641] Key type ._llcrypt registered [ 975.562629] Key type .llcrypt registered [ 975.658450] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 976.668849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 976.700264] alg: No test for adler32 (adler32-zlib) [ 977.713384] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 977.908771] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 979.528193] Key type lgssc registered [ 980.516066] Lustre: Echo OBD driver; http://www.lustre.org/ [ 987.666807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 987.678215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 989.008125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.291304] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 998.370241] LustreError: 30054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 998.387588] LustreError: 30054:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1001.091129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.641656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1005.693937] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1012.392477] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1018.144372] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1025.131131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1025.554341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1031.883973] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1033.843043] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1039.802330] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1057.249579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1057.266807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1057.297153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1058.427894] Lustre: server umount lustre-OST0000 complete [ 1064.929462] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1064.950833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1064.958061] Lustre: Skipped 1 previous similar message [ 1070.050422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1070.057304] Lustre: Skipped 1 previous similar message [ 1074.265418] Lustre: server umount lustre-MDT0000 complete [ 1075.173754] LustreError: 31307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1075.184174] LustreError: 31307:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1077.485488] LustreError: 30034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787373933 with bad export cookie 4453116974881549805 [ 1077.503660] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1077.504823] LustreError: 30034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1077.713852] Lustre: server umount lustre-MDT0001 complete [ 1088.761821] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 1091.470838] Key type lgssc unregistered [ 1091.856739] LNet: 33791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1091.873233] LNetError: 33791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1091.909963] LNet: Removed LNI 192.168.202.118@tcp [ 1092.751461] Key type .llcrypt unregistered [ 1092.753870] Key type ._llcrypt unregistered [ 1111.016448] Key type ._llcrypt registered [ 1111.018630] Key type .llcrypt registered [ 1111.088848] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1112.281870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1112.442571] alg: No test for adler32 (adler32-zlib) [ 1113.680816] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 1113.956552] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 1115.656187] Key type lgssc registered [ 1116.794882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1124.497528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1124.512589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1125.933636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1130.070322] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1136.102068] LustreError: 35063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1136.128381] LustreError: 35063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1137.886541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1138.328762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1142.655933] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1151.197777] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1157.612059] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1165.015914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1165.426065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1170.550607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1171.524082] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1178.526845] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1196.515105] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1196.523678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.541409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1198.564827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1198.583345] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1198.591843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1200.222788] Lustre: server umount lustre-OST0000 complete [ 1204.705550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1204.716594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1204.729281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1204.738796] Lustre: Skipped 1 previous similar message [ 1208.802786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1208.803986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1210.172868] Lustre: server umount lustre-MDT0000 complete [ 1213.925430] LustreError: 36684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1213.953955] LustreError: 36684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1214.070143] LustreError: 35041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787374069 with bad export cookie 6909160444301557281 [ 1214.072255] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1214.090782] LustreError: 35041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1214.471706] Lustre: server umount lustre-MDT0001 complete [ 1224.193346] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 1226.895306] Key type lgssc unregistered [ 1227.202407] LNet: 38620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1227.213666] LNetError: 38620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1227.258943] LNet: Removed LNI 192.168.202.118@tcp [ 1228.048995] Key type .llcrypt unregistered [ 1228.051493] Key type ._llcrypt unregistered [ 1252.063778] Key type ._llcrypt registered [ 1252.068536] Key type .llcrypt registered [ 1252.291631] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1253.822147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1254.021024] alg: No test for adler32 (adler32-zlib) [ 1255.112760] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 1255.434886] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 1257.153594] Key type lgssc registered [ 1258.365077] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1266.840648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1266.877046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1268.391816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1273.353394] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1278.950880] LustreError: 40035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1278.978951] LustreError: 40035:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1282.418399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1282.780045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1287.213938] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1295.324471] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1301.045308] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1307.385636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1307.562702] LustreError: 41857:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1307.713373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1313.038917] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1313.772693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1320.021291] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1326.936787] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.18@tcp (stopping) [ 1327.881963] Lustre: server umount lustre-MDT0000 complete [ 1329.121357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1329.143574] Lustre: Skipped 2 previous similar messages [ 1329.147085] LustreError: 40030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1329.168602] LustreError: 40030:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1331.596283] LustreError: 40015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787374187 with bad export cookie 1783037723828178910 [ 1331.600706] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1331.611385] LustreError: 40015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1332.045687] LustreError: 42526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1332.078430] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.18@tcp (stopping) [ 1334.243125] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1334.252866] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1336.295260] LustreError: 40031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1336.296163] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1336.316135] LustreError: 40031:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1338.137171] Lustre: server umount lustre-MDT0001 complete [ 1359.163180] Lustre: server umount lustre-OST0000 complete [ 1374.230157] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 1376.916108] Key type lgssc unregistered [ 1377.249876] LNet: 43901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1377.270858] LNetError: 43901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1377.327531] LNet: Removed LNI 192.168.202.118@tcp [ 1378.119668] Key type .llcrypt unregistered [ 1378.121284] Key type ._llcrypt unregistered [ 1401.896155] Key type ._llcrypt registered [ 1401.899528] Key type .llcrypt registered [ 1402.046959] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1403.461767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1403.480483] alg: No test for adler32 (adler32-zlib) [ 1404.616388] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 1404.791036] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 1406.443256] Key type lgssc registered [ 1407.612311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1414.800877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1414.811146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1416.294975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.238810] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1426.917699] LustreError: 45316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1426.941471] LustreError: 45316:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1429.767377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1430.131504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1434.373531] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1442.227061] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1447.553989] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1454.038726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1454.459028] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1460.823351] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1461.621359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1468.841198] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1476.066868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1476.075431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1476.091496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1477.091543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1477.100594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1477.102512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1482.233380] Lustre: server umount lustre-OST0000 complete [ 1501.538746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1501.547578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1501.552635] Lustre: Skipped 1 previous similar message [ 1501.556241] Lustre: Skipped 2 previous similar messages [ 1505.864507] Lustre: server umount lustre-MDT0000 complete [ 1506.794272] LustreError: 45317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1509.420286] LustreError: 45296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787374365 with bad export cookie 8832725044367766268 [ 1509.438684] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1515.750255] Lustre: server umount lustre-MDT0001 complete [ 1527.322700] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 1529.783632] Key type lgssc unregistered [ 1530.048650] LNet: 49029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.055241] LNetError: 49029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1530.074473] LNet: Removed LNI 192.168.202.118@tcp [ 1530.674394] Key type .llcrypt unregistered [ 1530.677091] Key type ._llcrypt unregistered [ 1552.909455] Key type ._llcrypt registered [ 1552.912629] Key type .llcrypt registered [ 1553.082384] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1554.035629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1554.094067] alg: No test for adler32 (adler32-zlib) [ 1555.107522] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 1555.313041] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 1557.000183] Key type lgssc registered [ 1558.224655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1565.754152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1565.804230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1567.281453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1572.193436] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1576.422247] LustreError: 50455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1576.445692] LustreError: 50455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1579.646444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1579.947876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1583.315773] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1590.903663] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1595.985809] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1601.626540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1601.893797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1607.710119] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1612.157105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1615.296988] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1621.038580] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1624.454396] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1636.329927] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-22 0h54m49s [ 1637.626441] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1641.446902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1641.461706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1641.476160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1642.547465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1642.550316] Lustre: Skipped 4 previous similar messages [ 1643.905605] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.18@tcp (stopping) [ 1643.915288] Lustre: Skipped 1 previous similar message [ 1645.581666] Lustre: server umount lustre-MDT0000 complete [ 1648.611517] LustreError: 51182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1648.630680] LustreError: 51182:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1648.790406] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1654.436397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1654.535726] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1654.705782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1654.752659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1655.735096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1658.440662] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1661.265352] Lustre: DEBUG MARKER: Wait for df (36731) ... [ 1662.600341] Lustre: DEBUG MARKER: done [ 1666.018718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1666.023909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1666.031056] Lustre: Skipped 3 previous similar messages [ 1666.045951] Lustre: Skipped 1 previous similar message [ 1671.150715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1671.158558] Lustre: Skipped 1 previous similar message [ 1671.756264] Lustre: server umount lustre-OST0000 complete [ 1676.262524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1676.264833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1676.283715] Lustre: Skipped 1 previous similar message [ 1682.151848] Lustre: server umount lustre-MDT0000 complete [ 1682.407056] LustreError: 51182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1682.434040] LustreError: 51182:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1686.023858] LustreError: 50434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787374541 with bad export cookie 9073175327083403493 [ 1686.041584] LustreError: 50434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1686.050958] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1686.320596] Lustre: server umount lustre-MDT0001 complete [ 1697.665687] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 1700.197607] Key type lgssc unregistered [ 1700.597155] LNet: 55385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1700.610255] LNetError: 55385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1700.628862] LNet: Removed LNI 192.168.202.118@tcp [ 1701.387435] Key type .llcrypt unregistered [ 1701.389539] Key type ._llcrypt unregistered [ 1736.241544] Key type ._llcrypt registered [ 1736.245087] Key type .llcrypt registered [ 1736.330499] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1737.325156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1737.334795] alg: No test for adler32 (adler32-zlib) [ 1738.298733] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 1738.486493] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 1740.096186] Key type lgssc registered [ 1741.007856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1748.237148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1748.251321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1749.502535] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1749.521652] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1749.695175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1753.974940] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1762.011235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1762.086273] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1762.118347] Lustre: 57592:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1762.325734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1767.014395] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1774.344687] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1779.469815] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1785.682524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1785.934392] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1786.099043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1791.874567] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1795.791944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1798.853747] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1805.601704] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-22 0h57m39s [ 1807.071787] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1813.843663] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1816.524609] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1816.582643] LustreError: 59459:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1836.899618] Lustre: lustre-MDT0000: Client 97b408c7-0167-4461-bffb-a580c0d90c82 (at 192.168.202.18@tcp) reconnecting [ 1856.585227] LustreError: 59459:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1859.007784] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1861.326543] Lustre: DEBUG MARKER: done [ 1872.878201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1872.889160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1872.922429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1875.571360] Lustre: server umount lustre-OST0000 complete [ 1883.118109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1883.139620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1883.144719] Lustre: Skipped 2 previous similar messages [ 1883.165612] Lustre: Skipped 2 previous similar messages [ 1884.193786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1884.205609] Lustre: Skipped 1 previous similar message [ 1890.275534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1890.287374] Lustre: Skipped 1 previous similar message [ 1894.880066] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1895.211920] Lustre: server umount lustre-MDT0000 complete [ 1899.286467] LustreError: 57584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787374755 with bad export cookie 13519903516041011728 [ 1899.290483] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1899.300376] LustreError: 57584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1899.954205] Lustre: server umount lustre-MDT0001 complete [ 1910.716835] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 1913.258399] Key type lgssc unregistered [ 1913.556747] LNet: 61929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1913.577573] LNetError: 61929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1913.608167] LNet: Removed LNI 192.168.202.118@tcp [ 1914.373633] Key type .llcrypt unregistered [ 1914.375270] Key type ._llcrypt unregistered [ 1961.130101] Key type ._llcrypt registered [ 1961.131591] Key type .llcrypt registered [ 1961.248282] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1962.618900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1962.645959] alg: No test for adler32 (adler32-zlib) [ 1963.727723] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 1963.999666] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 1965.664472] Key type lgssc registered [ 1966.964607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1970.563059] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1970.588183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.916650] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1971.971291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1972.197366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1973.200749] Lustre: Failing over lustre-MDT0000 [ 1973.502737] Lustre: server umount lustre-MDT0000 complete [ 1981.439721] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:00:35 (1787374835) [ 1988.879468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1989.429131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1994.412648] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2003.093064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2003.186820] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2003.261575] Lustre: 64730:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2003.509661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2008.484408] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2016.280794] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2021.777974] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2028.103450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2028.288679] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2028.545463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2034.323677] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2035.870472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2041.308452] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2046.652961] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2052.225314] Lustre: server umount lustre-MDT0000 complete [ 2054.634327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2054.653112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.546674] LustreError: 64723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787374911 with bad export cookie 15187738504194685737 [ 2055.557840] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2055.559109] LustreError: 64723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2056.011780] Lustre: server umount lustre-MDT0001 complete [ 2060.238599] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2064.719747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2072.864476] Lustre: 63652:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787374912/real 1787374912] req@00000000d6f07356 x1874198338246016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787374928 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2072.908682] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2073.648925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2076.896149] Lustre: 63655:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787374916/real 1787374916] req@0000000008249b98 x1874198338246144/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787374932 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2076.923955] Lustre: 63655:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2081.479266] LustreError: 66591:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2081.500108] LustreError: 66591:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2081.616944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2081.732318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2082.600344] LustreError: 68511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2082.614797] LustreError: 68511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2082.622266] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2083.623934] Lustre: 63653:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787374921/real 1787374921] req@00000000133f75cb x1874198338246528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787374937 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2083.650433] Lustre: 63653:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2086.527911] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2086.692119] Lustre: 63654:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787374926/real 1787374926] req@00000000c020640d x1874198338246912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787374942 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2086.740710] Lustre: 63654:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2087.909538] LustreError: 66591:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2087.932342] LustreError: 66591:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2093.026932] LustreError: 68509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2093.046942] LustreError: 68509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2096.051623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2096.335364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2101.242455] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2101.748478] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2110.360858] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2117.427419] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2127.329744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2127.337620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2127.344361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2127.345287] Lustre: Skipped 3 previous similar messages [ 2131.749510] Lustre: server umount lustre-MDT0000 complete [ 2132.457644] LustreError: 69062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2132.480398] LustreError: 69062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2136.356691] LustreError: 66597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787374992 with bad export cookie 15187738504194688012 [ 2136.361892] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2136.374902] LustreError: 66597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2137.120533] Lustre: server umount lustre-MDT0001 complete [ 2145.086574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2153.952122] Lustre: 63655:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787374993/real 1787374993] req@00000000d3eb8ef1 x1874198338274688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787375009 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2153.985067] Lustre: 63655:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2153.994824] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2154.099357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2162.631609] LustreError: 69246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2162.705379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2162.790574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2163.681119] Lustre: 63652:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787375002/real 1787375002] req@0000000034f42f77 x1874198338275200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787375018 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2163.709899] Lustre: 63652:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2167.190167] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2175.990559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2180.939905] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2181.617868] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2187.798557] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2193.166819] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2199.817518] Lustre: server umount lustre-MDT0000 complete [ 2202.085296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2202.093729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2202.112651] LustreError: 66591:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2202.125057] LustreError: 66591:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2203.090969] LustreError: 64722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787375058 with bad export cookie 15187738504194689391 [ 2203.093062] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2203.099452] LustreError: 64722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2203.125264] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2203.132508] Lustre: Skipped 2 previous similar messages [ 2205.517129] Lustre: server umount lustre-MDT0001 complete [ 2211.483573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2215.288159] Lustre: DEBUG MARKER: files compared the same [ 2217.287958] Lustre: server umount lustre-OST0000 complete [ 2233.654516] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 2236.157948] Key type lgssc unregistered [ 2236.399863] LNet: 74470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2236.404126] LNetError: 74470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2236.415873] LNet: Removed LNI 192.168.202.118@tcp [ 2237.146380] Key type .llcrypt unregistered [ 2237.151751] Key type ._llcrypt unregistered [ 2259.846246] Key type ._llcrypt registered [ 2259.848640] Key type .llcrypt registered [ 2259.948704] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2261.198089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2261.261615] alg: No test for adler32 (adler32-zlib) [ 2262.345434] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 2262.661366] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 2264.440170] Key type lgssc registered [ 2265.831020] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2273.622248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2273.642377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2275.030843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2279.107917] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug +malloc all [ 2284.515990] LustreError: 75889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2284.536424] LustreError: 75889:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2287.480338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2287.680295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2291.820950] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug +malloc all [ 2298.541521] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2304.154821] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2312.230898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2312.523727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2319.720497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2320.124732] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug +malloc all [ 2329.073766] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2339.297061] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2339.302125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2339.310647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2340.320867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2340.325306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2344.517352] Lustre: server umount lustre-OST0000 complete [ 2349.025296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2349.043932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2349.060514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2349.063201] Lustre: Skipped 1 previous similar message [ 2354.787802] Lustre: server umount lustre-MDT0000 complete [ 2355.680841] LustreError: 75889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2355.691892] LustreError: 75889:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2358.493290] LustreError: 75868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787375214 with bad export cookie 15975053803153394362 [ 2358.494198] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2358.503713] LustreError: 75868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2358.709505] Lustre: server umount lustre-MDT0001 complete [ 2368.387371] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 2371.059331] Key type lgssc unregistered [ 2371.291714] LNet: 79424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2371.297270] LNetError: 79424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2371.316748] LNet: Removed LNI 192.168.202.118@tcp [ 2372.091866] Key type .llcrypt unregistered [ 2372.094860] Key type ._llcrypt unregistered [ 2393.837610] Key type ._llcrypt registered [ 2393.840146] Key type .llcrypt registered [ 2393.925959] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2395.234584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2395.277125] alg: No test for adler32 (adler32-zlib) [ 2396.235870] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 2396.409896] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 2398.040688] Key type lgssc registered [ 2399.100539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2406.477892] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2406.502861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2422.049389] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2427.168466] LustreError: 80818:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.118@tcp: failed processing log, type 4: rc = -110 [ 2453.856689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2460.692991] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2469.263696] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2477.648433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2477.883670] Lustre: *** cfs_fail_loc=706, val=0*** [ 2478.157686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2478.220776] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2482.757327] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2483.184217] LustreError: 80844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2483.195673] LustreError: 80844:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2488.294933] LustreError: 80843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2488.315239] LustreError: 80843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2490.603143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.007670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2495.242537] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2502.544420] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2508.370769] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2511.332377] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2511.346292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2511.369833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2516.488674] Lustre: server umount lustre-OST0000 complete [ 2522.081695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2522.085710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2522.094596] Lustre: Skipped 2 previous similar messages [ 2522.120496] Lustre: Skipped 2 previous similar messages [ 2527.096255] Lustre: server umount lustre-MDT0000 complete [ 2527.203842] LustreError: 81903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2527.222419] LustreError: 81903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2530.436913] LustreError: 80824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787375386 with bad export cookie 18072465998128425113 [ 2530.447712] LustreError: 80824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2530.449290] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2530.706569] Lustre: server umount lustre-MDT0001 complete [ 2540.420334] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 2542.898251] Key type lgssc unregistered [ 2543.140692] LNet: 84435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2543.146891] LNetError: 84435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2543.163340] LNet: Removed LNI 192.168.202.118@tcp [ 2543.908381] Key type .llcrypt unregistered [ 2543.910912] Key type ._llcrypt unregistered [ 2570.049899] Key type ._llcrypt registered [ 2570.052357] Key type .llcrypt registered [ 2570.173841] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2571.617032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2571.668000] alg: No test for adler32 (adler32-zlib) [ 2572.805744] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 2573.085188] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 2574.760189] Key type lgssc registered [ 2575.613853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2581.798189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2581.820657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2587.285827] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2596.147543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2596.740958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2601.869409] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2601.961475] LustreError: 86540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.971641] LustreError: 86540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2607.082302] LustreError: 86543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2607.110637] LustreError: 86543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2611.064915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2611.359313] LustreError: 86540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.374266] LustreError: 86540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2611.383699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2616.701977] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2616.807924] LustreError: 87289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2616.837275] LustreError: 87289:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2621.925389] LustreError: 86544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2621.945973] LustreError: 86544:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2625.431407] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2625.449772] Lustre: MGS: Client 960196b8-1315-4cad-8ee0-d5c2748458bd (at 0@lo) reconnecting [ 2625.483718] LustreError: 86554:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2625.692737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2625.756261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2629.713525] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2646.500525] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2646.511079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2646.541471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2648.923691] Lustre: server umount lustre-OST0000 complete [ 2656.738149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2656.755658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2656.757080] Lustre: Skipped 2 previous similar messages [ 2656.770551] Lustre: Skipped 2 previous similar messages [ 2658.900543] Lustre: server umount lustre-MDT0000 complete [ 2661.862363] LustreError: 88085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2661.876283] LustreError: 88085:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2663.100154] Lustre: server umount lustre-MDT0001 complete [ 2667.154411] Lustre: server umount MGS complete [ 2676.672927] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:12:10 (1787375530) [ 2700.566385] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 2710.461305] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2743.402577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2747.427882] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2756.682136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2756.944268] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2756.990791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2757.060304] Lustre: lustre-MDT0001: new disk, initializing [ 2757.150824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2757.168286] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2757.178788] Lustre: 93707:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2757.188559] LustreError: 93707:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2761.674845] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2772.466504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2772.765582] Lustre: lustre-OST0000: new disk, initializing [ 2772.769464] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2772.775193] Lustre: 94473:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2772.858508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2779.050366] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2780.193153] Lustre: 94833:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000925ed831 x1874198976830464/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787375651 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2789.301164] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2796.521731] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2796.544684] Lustre: MGS: Client 7e1fbc36-a27d-4f52-a8c0-d48c1f3aeba2 (at 0@lo) reconnecting [ 2796.609522] Lustre: 92977:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2796.639801] LustreError: 92977:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2796.650154] LustreError: 92977:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2796.701595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2796.838864] Lustre: lustre-MDT0000: new disk, initializing [ 2796.968532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2796.988391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2797.004375] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2798.051643] Lustre: 95543:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004dabf9a6 x1874198976837376/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787375669 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2801.420990] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2803.168663] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2807.146227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2807.155619] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2807.232646] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2832.870636] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2832.875160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2832.885242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2838.298768] Lustre: server umount lustre-OST0000 complete [ 2844.644885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2844.645396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2844.654443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2844.654451] Lustre: Skipped 3 previous similar messages [ 2844.661556] Lustre: Skipped 2 previous similar messages [ 2844.677252] LustreError: Skipped 1 previous similar message [ 2848.468789] Lustre: server umount lustre-MDT0000 complete [ 2849.761445] LustreError: 93679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2849.787862] LustreError: 93679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2858.982709] Lustre: server umount lustre-MDT0001 complete [ 2863.479517] Lustre: server umount MGS complete [ 2876.278415] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 01:15:29 (1787375729) [ 2893.016950] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 2896.131876] Key type lgssc unregistered [ 2896.374888] LNet: 98054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2896.379836] LNetError: 98054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2897.448387] LNet: Removed LNI 192.168.202.118@tcp [ 2898.320374] Key type .llcrypt unregistered [ 2898.321542] Key type ._llcrypt unregistered [ 2914.391240] Key type ._llcrypt registered [ 2914.392753] Key type .llcrypt registered [ 2914.486583] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2915.368517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2915.444320] alg: No test for adler32 (adler32-zlib) [ 2916.505171] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 2916.761571] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 2918.416203] Key type lgssc registered [ 2919.447554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2926.128172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2926.149865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.730344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2932.133616] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2937.315442] LustreError: 99286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2937.336963] LustreError: 99286:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2940.391651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2940.819301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2945.148630] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2951.281657] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2956.328485] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2962.658561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2963.000981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2968.892904] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2974.251612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2976.765292] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2986.468626] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2986.481447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2986.509614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2989.537839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2989.549818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2989.566036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2991.781658] Lustre: server umount lustre-OST0000 complete [ 2997.219089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2997.228940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2997.247884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2997.250978] Lustre: Skipped 1 previous similar message [ 2999.801315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2999.804545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3002.129872] Lustre: server umount lustre-MDT0000 complete [ 3004.897768] LustreError: 99286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3004.910437] LustreError: 99286:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3006.760157] LustreError: 99267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787375862 with bad export cookie 6462002869591003227 [ 3006.764576] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3006.980224] Lustre: server umount lustre-MDT0001 complete [ 3022.260900] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 3025.411288] Key type lgssc unregistered [ 3025.888719] LNet: 102822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3025.902938] LNetError: 102822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3025.922630] LNet: Removed LNI 192.168.202.118@tcp [ 3026.846796] Key type .llcrypt unregistered [ 3026.848789] Key type ._llcrypt unregistered [ 3044.144155] Key type ._llcrypt registered [ 3044.151929] Key type .llcrypt registered [ 3044.402452] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3045.624928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3045.788569] alg: No test for adler32 (adler32-zlib) [ 3046.878880] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 3047.110848] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 3048.776276] Key type lgssc registered [ 3050.029989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3056.513969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3056.524832] LustreError: 103903:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3056.533619] LustreError: 103906:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3056.541385] LustreError: 103903:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4990 [ 3056.547437] LustreError: 103903:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3056.560882] LustreError: 103903:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3056.575691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3058.136931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3066.474355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3066.922681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3071.854793] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3078.371529] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing lsmod [ 3081.021814] LustreError: 105113:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3081.030758] LustreError: 105114:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3081.037696] LustreError: 105113:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4990 [ 3081.040634] LustreError: 105113:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3081.044179] LustreError: 105113:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3081.054820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3081.285477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3082.362049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3087.342559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3087.365392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3087.388516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3090.590511] Lustre: server umount lustre-MDT0000 complete [ 3091.941498] LustreError: 103944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3091.959528] LustreError: 103944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3095.192786] LustreError: 104383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787375950 with bad export cookie 1301024260545941713 [ 3095.194808] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3095.202414] LustreError: 104383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3096.096320] LustreError: 104376:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000004f4acd4a x1874199473836800/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3096.596292] Lustre: server umount lustre-MDT0001 complete [ 3112.023916] Lustre: server umount lustre-OST0000 complete [ 3122.855482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3123.224877] LustreError: 106178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3123.292756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3127.633507] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3128.297366] LustreError: 106179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3133.414344] LustreError: 106178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3135.968926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3136.260856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3140.288771] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3147.883977] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3153.740978] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3160.612223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3160.968030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3168.698611] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3169.144290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3178.134935] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3187.662629] Lustre: server umount lustre-OST0000 complete [ 3192.802330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3192.812041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3192.837603] Lustre: Skipped 2 previous similar messages [ 3192.844593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3192.860718] Lustre: Skipped 2 previous similar messages [ 3194.852294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3194.853323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3199.969894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3199.983443] Lustre: Skipped 2 previous similar messages [ 3203.875085] Lustre: 103307:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787376043/real 1787376043] req@00000000dbf5310f x1874199473865856/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787376059 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3203.896500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3206.112233] Lustre: 103309:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787376045/real 1787376045] req@0000000023c24a5a x1874199473866624/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787376061 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3206.114127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3206.354274] Lustre: server umount lustre-MDT0000 complete [ 3209.193135] LustreError: 106175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3209.212030] LustreError: 106175:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3210.373324] LustreError: 106159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787376066 with bad export cookie 1301024260545942637 [ 3210.386054] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3210.397078] LustreError: 106159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3211.179248] Lustre: server umount lustre-MDT0001 complete [ 3223.306402] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 3226.352748] Key type lgssc unregistered [ 3226.613532] LNet: 109716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3226.620595] LNetError: 109716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3226.658118] LNet: Removed LNI 192.168.202.118@tcp [ 3227.702217] Key type .llcrypt unregistered [ 3227.704211] Key type ._llcrypt unregistered [ 3255.083858] Key type ._llcrypt registered [ 3255.086731] Key type .llcrypt registered [ 3255.235671] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3256.939081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3256.980588] alg: No test for adler32 (adler32-zlib) [ 3258.054717] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 3258.264116] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 3259.936291] Key type lgssc registered [ 3261.270883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3270.156636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3270.184477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3271.806299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3277.103603] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3281.895632] LustreError: 111139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3281.910011] LustreError: 111139:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3285.136908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3285.470549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3290.538866] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3298.184474] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3304.362202] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3311.469560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3311.851139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3316.983593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3318.131805] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3326.189360] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3335.521750] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3339.559828] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3345.905599] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3345.921343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3345.922195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3345.955620] Lustre: Skipped 1 previous similar message [ 3345.974928] Lustre: Skipped 1 previous similar message [ 3351.011758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3351.020623] Lustre: Skipped 1 previous similar message [ 3351.757414] Lustre: server umount lustre-OST0000 complete [ 3357.154943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3357.166405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3357.188072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3361.251724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3361.260060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3362.942384] Lustre: server umount lustre-MDT0000 complete [ 3366.377667] LustreError: 112678:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3366.405702] LustreError: 112678:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3367.487076] LustreError: 111875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787376223 with bad export cookie 8968764887528221803 [ 3367.494424] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3367.496730] LustreError: 111875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3368.086907] Lustre: server umount lustre-MDT0001 complete [ 3378.745672] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 3381.705870] Key type lgssc unregistered [ 3381.960768] LNet: 114781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3381.964334] LNetError: 114781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3381.978058] LNet: Removed LNI 192.168.202.118@tcp [ 3382.975932] Key type .llcrypt unregistered [ 3382.986910] Key type ._llcrypt unregistered [ 3404.013336] Key type ._llcrypt registered [ 3404.014980] Key type .llcrypt registered [ 3404.100645] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3405.392875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3405.448742] alg: No test for adler32 (adler32-zlib) [ 3406.670545] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 3406.924867] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 3408.624183] Key type lgssc registered [ 3409.959960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3419.629827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3419.647963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3421.244671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3427.221402] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3430.380544] LustreError: 116007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3430.400547] LustreError: 116007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3435.510565] LustreError: 116006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3438.041199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3438.532668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3443.813960] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3452.590321] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3459.649894] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3467.993628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3468.372428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3475.373671] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3483.349487] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3490.951560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3504.617292] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3504.646176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3504.673598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3506.658985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3506.676133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3506.679611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3510.626464] Lustre: server umount lustre-OST0000 complete [ 3516.898231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3516.901205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3516.932194] Lustre: Skipped 1 previous similar message [ 3516.959502] Lustre: Skipped 2 previous similar messages [ 3522.019219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3522.869326] Lustre: server umount lustre-MDT0000 complete [ 3527.138628] LustreError: 116001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3527.163819] LustreError: 116001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3527.586202] LustreError: 115986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787376383 with bad export cookie 14128233060847529401 [ 3527.594304] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3527.964555] Lustre: server umount lustre-MDT0001 complete [ 3541.777529] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 3545.197423] Key type lgssc unregistered [ 3545.419903] LNet: 119565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3545.423893] LNetError: 119565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3545.434854] LNet: Removed LNI 192.168.202.118@tcp [ 3546.580662] Key type .llcrypt unregistered [ 3546.582416] Key type ._llcrypt unregistered [ 3574.691325] Key type ._llcrypt registered [ 3574.693577] Key type .llcrypt registered [ 3574.844975] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3576.049861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3576.202391] alg: No test for adler32 (adler32-zlib) [ 3577.348954] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 3577.595858] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 3579.348348] Key type lgssc registered [ 3580.638458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3589.464856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3589.492211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3591.033508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3595.435650] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3600.358682] LustreError: 120986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3600.387557] LustreError: 120986:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3605.477121] LustreError: 120987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3605.577398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3605.983281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3611.373852] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3621.981601] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3628.702157] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3636.773056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3637.263277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3644.052805] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3653.252051] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3661.818952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3670.366830] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3674.848343] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3674.859445] Lustre: Skipped 2 previous similar messages [ 3677.310460] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3677.352267] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3679.716474] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3679.725839] Lustre: Skipped 2 previous similar messages [ 3688.492567] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3688.501359] Lustre: Skipped 1 previous similar message [ 3694.047734] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.18@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3694.055278] Lustre: Skipped 2 previous similar messages [ 3702.993643] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.18@tcp 192.168.0.[2,10]@tcp [ 3707.102836] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.18/24@tcp in log lustre-MDT0000 [ 3707.108046] Lustre: Skipped 2 previous similar messages [ 3709.602559] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.18/24@tcp [ 3709.609429] Lustre: Skipped 1 previous similar message [ 3721.193446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3721.193537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3721.205853] LustreError: Skipped 3 previous similar messages [ 3721.222588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3722.523689] Lustre: server umount lustre-OST0000 complete [ 3728.359956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3728.376197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3728.378709] Lustre: Skipped 1 previous similar message [ 3728.395150] Lustre: Skipped 1 previous similar message [ 3733.474283] LustreError: 120986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3733.533494] Lustre: server umount lustre-MDT0000 complete [ 3737.515236] LustreError: 124493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787376593 with bad export cookie 6471204002038247552 [ 3737.523771] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3737.530426] LustreError: 124493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3747.952282] Lustre: server umount lustre-MDT0001 complete [ 3760.295243] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 3763.268697] Key type lgssc unregistered [ 3763.619682] LNet: 126054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3763.629712] LNetError: 126054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3763.646234] LNet: Removed LNI 192.168.202.118@tcp [ 3764.731235] Key type .llcrypt unregistered [ 3764.739477] Key type ._llcrypt unregistered [ 3800.581809] Key type ._llcrypt registered [ 3800.583595] Key type .llcrypt registered [ 3800.700235] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3802.480528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3802.499728] alg: No test for adler32 (adler32-zlib) [ 3803.522593] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 3803.874100] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 3805.560200] Key type lgssc registered [ 3806.916817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3814.753248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3814.775761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3816.397743] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3816.419947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3816.423786] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.18/24@tcp [ 3821.204222] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3826.658603] LustreError: 127620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3826.667363] LustreError: 127620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3830.609969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3830.966911] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3831.016264] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.18/24@tcp [ 3831.092183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3836.516625] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3845.668234] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3854.306418] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3863.070183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3863.609899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3870.966971] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3881.216758] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3887.278415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3901.410319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3901.430565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3901.464083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3902.951633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3902.952470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3902.985796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3908.068627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3908.080117] Lustre: Skipped 2 previous similar messages [ 3913.189607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3913.203400] Lustre: Skipped 1 previous similar message [ 3914.208170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3914.430434] Lustre: server umount lustre-OST0000 complete [ 3923.429560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3923.434477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3923.442438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3923.448096] Lustre: Skipped 1 previous similar message [ 3925.828559] Lustre: server umount lustre-MDT0000 complete [ 3928.547728] LustreError: 127619:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3928.570434] LustreError: 127619:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3930.384974] LustreError: 127601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787376786 with bad export cookie 17462160092779249256 [ 3930.407567] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3930.892543] Lustre: server umount lustre-MDT0001 complete [ 3944.697493] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 3948.096951] Key type lgssc unregistered [ 3948.480741] LNet: 131238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3948.488606] LNetError: 131238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3948.506692] LNet: Removed LNI 192.168.202.118@tcp [ 3949.489899] Key type .llcrypt unregistered [ 3949.492381] Key type ._llcrypt unregistered [ 3978.766183] Key type ._llcrypt registered [ 3978.769641] Key type .llcrypt registered [ 3978.897832] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 3995.552069] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3996.782554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3996.812121] alg: No test for adler32 (adler32-zlib) [ 3998.039564] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 3998.307577] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 4000.016286] Key type lgssc registered [ 4001.649942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4037.482602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4037.498967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4038.813082] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4038.835311] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4038.890654] Lustre: lustre-MDT0000: new disk, initializing [ 4038.947567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4038.960967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4043.058550] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4053.610626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4053.791721] Lustre: 134638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4053.856075] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4053.862956] Lustre: Skipped 1 previous similar message [ 4053.944992] Lustre: lustre-MDT0001: new disk, initializing [ 4054.054270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4054.091297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4054.102371] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4058.880830] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4068.141377] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4074.236783] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4080.961184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4081.207220] Lustre: lustre-OST0000: new disk, initializing [ 4081.209394] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4081.214432] Lustre: 136609:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4081.321614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4087.679162] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4087.884526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4087.891732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4088.033193] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4096.658788] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4100.977843] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4101.175545] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4106.197861] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4106.404756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4108.771263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4108.792240] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4108.806735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4113.897762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4113.909781] Lustre: Skipped 1 previous similar message [ 4114.660989] Lustre: server umount lustre-OST0000 complete [ 4119.014560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4119.016042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4119.022755] Lustre: Skipped 2 previous similar messages [ 4119.034200] Lustre: Skipped 2 previous similar messages [ 4124.132242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4124.134780] Lustre: Skipped 1 previous similar message [ 4124.552994] Lustre: server umount lustre-MDT0000 complete [ 4128.144964] LustreError: 134629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787376983 with bad export cookie 4320011765290498968 [ 4128.161320] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4128.168489] LustreError: 134629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4128.890557] Lustre: server umount lustre-MDT0001 complete [ 4137.784302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4138.370632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4143.349628] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4143.595110] LustreError: 138825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4143.614575] LustreError: 138825:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4148.708545] LustreError: 138826:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4152.070373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4152.361292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4156.514329] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4164.071866] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4169.526560] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4175.485164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4175.876081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4181.293246] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4187.719373] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4191.761763] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4191.923899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4196.212486] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4196.372454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4205.262194] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4205.395228] Lustre: lustre-OST0001: new disk, initializing [ 4205.399978] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4205.405588] Lustre: 142196:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4205.496853] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4210.615715] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4214.868655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4214.876458] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4214.961974] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4218.247279] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4218.471541] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4223.095480] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4223.289709] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4229.408935] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f9c582e3800.ost_server_uuid 50 [ 4231.848427] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f9c582e3800.ost_server_uuid in FULL state after 0 sec [ 4245.988755] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4245.993774] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4246.003085] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4246.011185] Lustre: Skipped 1 previous similar message [ 4246.040472] Lustre: Skipped 1 previous similar message [ 4248.038916] Lustre: server umount lustre-OST0001 complete [ 4251.134728] LustreError: 140647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4251.198952] LustreError: 140647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4254.689522] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4254.694354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4254.707307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4256.236119] LustreError: 141159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4256.249272] LustreError: 141159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4258.480141] Lustre: server umount lustre-OST0000 complete [ 4264.929886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4264.934889] LustreError: Skipped 1 previous similar message [ 4264.939221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4264.945586] Lustre: Skipped 1 previous similar message [ 4268.902596] Lustre: server umount lustre-MDT0000 complete [ 4271.592244] LustreError: 139553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4271.613440] LustreError: 139553:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4273.238697] LustreError: 138805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787377128 with bad export cookie 4320011765290499850 [ 4273.241899] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4273.248810] LustreError: 138805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4273.555555] Lustre: server umount lustre-MDT0001 complete [ 4294.666037] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 4297.487316] Key type lgssc unregistered [ 4297.747554] LNet: 145202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4297.754765] LNetError: 145202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4297.775873] LNet: Removed LNI 192.168.202.118@tcp [ 4298.836760] Key type .llcrypt unregistered [ 4298.843027] Key type ._llcrypt unregistered [ 4339.690811] Key type ._llcrypt registered [ 4339.695762] Key type .llcrypt registered [ 4339.881483] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 4356.045454] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 4357.603133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4357.806984] alg: No test for adler32 (adler32-zlib) [ 4359.000581] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 4359.359036] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 4361.112088] Key type lgssc registered [ 4362.527360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4403.120688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4403.140260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4404.334779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4404.369959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4404.440882] Lustre: lustre-MDT0000: new disk, initializing [ 4404.515626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4404.539911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4408.443888] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4418.809733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4418.991761] Lustre: 149357:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4419.118578] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4419.124255] Lustre: Skipped 1 previous similar message [ 4419.200213] Lustre: lustre-MDT0001: new disk, initializing [ 4419.324442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4419.398040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4419.417154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4423.723972] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4433.765774] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4440.317989] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4446.673215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4446.898724] Lustre: lustre-OST0000: new disk, initializing [ 4446.902608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4446.907443] Lustre: 151329:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4446.996250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4452.449910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4452.464750] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4452.588015] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4453.385699] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4463.257633] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4468.270852] Lustre: Failing over lustre-OST0000 [ 4468.290467] LustreError: 152134:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0x2:0x0].0x0 (000000002eb95c0f) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4468.592933] Lustre: server umount lustre-OST0000 complete [ 4486.116824] Lustre: 147585:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787377325/real 1787377325] req@000000007f6434ee x1874200850630272/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787377341 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4486.170561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4487.987188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4488.359495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4488.388556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4488.408597] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4489.440123] Lustre: 147585:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787377329/real 1787377329] req@000000008bbb2031 x1874200850630784/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787377345 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4491.425723] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4494.356570] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4494.560120] Lustre: 147583:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787377334/real 1787377334] req@00000000c09c72a3 x1874200850632576/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787377350 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4494.591059] Lustre: 147583:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4496.259085] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4496.259576] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4497.798061] Lustre: Failing over lustre-MDT0000 [ 4498.231753] Lustre: server umount lustre-MDT0000 complete [ 4499.424250] Lustre: 147583:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787377339/real 1787377339] req@000000001c8ba4b9 x1874200850633600/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787377355 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4499.438890] Lustre: 147583:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4501.475057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4501.486286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4506.441988] LustreError: 150689:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4506.463766] LustreError: 150689:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4507.106773] LustreError: 149371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4507.149780] LustreError: 149371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4511.558749] LustreError: 149367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4516.679131] LustreError: 149366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4516.711867] LustreError: 149366:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4518.159917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4518.368544] Lustre: 147582:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787377357/real 1787377357] req@0000000039bff29a x1874200850644608/t0(0) o400->MGC192.168.202.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787377373 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4518.427246] Lustre: 147582:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4518.435624] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4518.856776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4518.922238] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4521.666228] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4523.472409] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4524.029994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4524.042675] Lustre: Skipped 1 previous similar message [ 4524.080175] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4524.135681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4532.193570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4532.196069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4532.211046] Lustre: Skipped 2 previous similar messages [ 4532.217350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4534.245524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4538.549108] Lustre: server umount lustre-OST0000 complete [ 4544.481603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4544.497065] Lustre: Skipped 1 previous similar message [ 4544.509978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4544.523652] Lustre: Skipped 2 previous similar messages [ 4548.472342] Lustre: server umount lustre-MDT0000 complete [ 4549.601603] LustreError: 150689:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4549.625994] LustreError: 150689:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4552.415107] LustreError: 149350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787377408 with bad export cookie 3405046075069776861 [ 4552.421198] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4552.424134] LustreError: 149350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4552.921168] Lustre: server umount lustre-MDT0001 complete [ 4565.337596] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 4568.889876] Key type lgssc unregistered [ 4569.222391] LNet: 154984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4569.238666] LNetError: 154984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4569.257316] LNet: Removed LNI 192.168.202.118@tcp [ 4570.253089] Key type .llcrypt unregistered [ 4570.257665] Key type ._llcrypt unregistered [ 4612.131524] Key type ._llcrypt registered [ 4612.134463] Key type .llcrypt registered [ 4612.258826] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 4613.638160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4613.750247] alg: No test for adler32 (adler32-zlib) [ 4614.970599] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 4615.370259] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 4617.073699] Key type lgssc registered [ 4618.464858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4626.062509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4626.086674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4627.598502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4632.981911] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4638.181991] LustreError: 156505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4638.216038] LustreError: 156505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4643.300381] LustreError: 156506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4644.177092] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4648.422975] LustreError: 156505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4648.554663] LustreError: 156534:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4648.571504] LustreError: 156534:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4648.581606] LustreError: 156534:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4648.994682] Lustre: server umount lustre-MDT0000 complete [ 4666.133201] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 4677.620523] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 4712.588401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4712.817920] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4712.854235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4712.937424] Lustre: lustre-MDT0000: new disk, initializing [ 4713.019857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4713.042884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4718.183508] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4729.263675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4729.417285] Lustre: 159840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4729.468081] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4729.471989] Lustre: Skipped 1 previous similar message [ 4729.572912] Lustre: lustre-MDT0001: new disk, initializing [ 4729.665495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4729.710250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4729.725949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4733.943941] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4744.207296] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4750.747668] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4758.007672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4758.260532] Lustre: lustre-OST0000: new disk, initializing [ 4758.264047] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4758.270099] Lustre: 161808:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4758.362283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4763.726460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4763.741265] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4763.867179] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4764.488893] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4774.127920] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4779.476954] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4781.325607] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 01:47:15 (1787377635) [ 5015.014841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5015.018392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5015.037108] Lustre: Skipped 1 previous similar message [ 5015.049091] Lustre: Skipped 2 previous similar messages [ 5020.136315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5020.140939] Lustre: Skipped 2 previous similar messages [ 5025.249479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5025.270936] Lustre: Skipped 2 previous similar messages [ 5026.272760] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5026.573304] Lustre: server umount lustre-MDT0000 complete [ 5030.369768] LustreError: 159847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5030.391769] LustreError: 159847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5030.787475] LustreError: 160633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787377886 with bad export cookie 18324601734248118131 [ 5030.788917] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5030.808869] LustreError: 160633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5035.489050] LustreError: 161209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5035.489419] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5035.539736] Lustre: Skipped 1 previous similar message [ 5035.546475] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5040.616766] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5045.216353] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5045.627233] Lustre: server umount lustre-MDT0001 complete [ 5055.594874] Lustre: server umount lustre-OST0000 complete [ 5065.260284] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 5076.140109] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 5115.161893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5115.468677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5115.523697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5115.611262] Lustre: lustre-MDT0000: new disk, initializing [ 5115.693341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5115.727217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5120.248967] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5132.722858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5132.856848] Lustre: 166471:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5132.891996] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5132.893888] Lustre: Skipped 1 previous similar message [ 5132.966871] Lustre: lustre-MDT0001: new disk, initializing [ 5133.064117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5133.136325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5133.149139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5138.478914] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5149.365768] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5155.094163] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5162.517140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5162.710922] Lustre: lustre-OST0000: new disk, initializing [ 5162.714972] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5162.722158] Lustre: 168440:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5162.811803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5168.215682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5168.228815] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5168.394060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5168.936669] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5179.099719] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5184.896573] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5185.105562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5189.458238] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5189.645099] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5194.212919] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5194.217359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5194.218352] LustreError: Skipped 1 previous similar message [ 5194.229741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5194.245864] Lustre: Skipped 1 previous similar message [ 5197.398804] Lustre: server umount lustre-OST0000 complete [ 5204.451281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5204.477599] Lustre: Skipped 2 previous similar messages [ 5207.404315] Lustre: server umount lustre-MDT0000 complete [ 5209.577208] LustreError: 167635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5209.603165] LustreError: 167635:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5211.135652] LustreError: 166461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378066 with bad export cookie 18324601734248276954 [ 5211.141342] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5211.144769] LustreError: 166461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5211.590746] Lustre: server umount lustre-MDT0001 complete [ 5220.710392] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:54:34 (1787378074) [ 5237.831839] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 5252.695727] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 5290.303844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5290.556369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5290.598805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5290.671571] Lustre: lustre-MDT0000: new disk, initializing [ 5290.752203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5290.790901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5295.303631] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5305.776923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5305.906274] Lustre: 173216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5306.108953] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5309.782973] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5319.869319] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5326.487784] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5333.359121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5333.781421] Lustre: lustre-OST0000: new disk, initializing [ 5333.786550] Lustre: Skipped 1 previous similar message [ 5333.790601] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5333.799615] Lustre: Skipped 2 previous similar messages [ 5333.821097] Lustre: 175195:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5339.263501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5339.278831] Lustre: Skipped 1 previous similar message [ 5339.290117] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5339.409551] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5340.750268] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5351.357545] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5360.097150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5360.100869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5360.112777] Lustre: Skipped 1 previous similar message [ 5360.113653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5360.120068] Lustre: Skipped 4 previous similar messages [ 5364.538569] Lustre: server umount lustre-OST0000 complete [ 5370.340288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5370.366064] Lustre: Skipped 1 previous similar message [ 5374.959183] Lustre: server umount lustre-MDT0000 complete [ 5375.459330] LustreError: 173229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5375.482890] LustreError: 173229:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5378.694613] LustreError: 175194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378234 with bad export cookie 18324601734248277843 [ 5378.699294] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5378.705524] LustreError: 175194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5379.031220] Lustre: server umount lustre-MDT0001 complete [ 5388.344701] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:57:22 (1787378242) [ 5405.648813] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 5415.642320] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 5453.864437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5454.129683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5454.158292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5454.245812] Lustre: lustre-MDT0000: new disk, initializing [ 5454.362926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5454.369330] Lustre: Skipped 2 previous similar messages [ 5454.398396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5459.015234] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5470.499267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5470.652886] Lustre: 179517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5470.952822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5475.137742] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5485.705809] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5492.364335] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5500.057390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5500.399584] Lustre: 181494:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5506.870840] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5510.199691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5510.294137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5516.781728] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5525.478680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5525.493488] LustreError: Skipped 1 previous similar message [ 5525.501295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5525.518481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5525.526807] Lustre: Skipped 2 previous similar messages [ 5538.272433] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5538.459822] Lustre: server umount lustre-OST0000 complete [ 5542.886050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5542.891952] LustreError: Skipped 1 previous similar message [ 5548.476157] Lustre: server umount lustre-MDT0000 complete [ 5551.589335] LustreError: 179523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5551.634424] LustreError: 179523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5553.506873] LustreError: 179509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378409 with bad export cookie 18324601734248278921 [ 5553.521308] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5553.532303] LustreError: 179509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5554.024357] Lustre: server umount lustre-MDT0001 complete [ 5569.668531] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 5572.503111] Key type lgssc unregistered [ 5572.753807] LNet: 183354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5572.757961] LNetError: 183354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5572.767876] LNet: Removed LNI 192.168.202.118@tcp [ 5573.417070] Key type .llcrypt unregistered [ 5573.422684] Key type ._llcrypt unregistered [ 5601.610399] Key type ._llcrypt registered [ 5601.612983] Key type .llcrypt registered [ 5601.732672] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 5602.711876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5602.866532] alg: No test for adler32 (adler32-zlib) [ 5604.108133] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 5604.458939] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 5606.320264] Key type lgssc registered [ 5607.938173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5616.567600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5616.591616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5618.136327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5622.772085] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5627.368511] LustreError: 184780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5627.397218] LustreError: 184780:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5632.486447] LustreError: 184781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5633.063568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5633.543879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5641.486391] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5650.336390] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5656.648381] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5663.302530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5663.707373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5668.562567] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5677.232870] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5687.279551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5707.757343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5707.762254] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5707.765521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5707.771256] Lustre: Skipped 1 previous similar message [ 5711.561545] Lustre: server umount lustre-OST0000 complete [ 5717.989624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5718.015794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5718.026849] Lustre: Skipped 2 previous similar messages [ 5722.654386] Lustre: server umount lustre-MDT0000 complete [ 5723.109700] LustreError: 184776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5723.135458] LustreError: 184776:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5726.831484] LustreError: 184763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378582 with bad export cookie 11432533018138009485 [ 5726.842939] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5726.850468] LustreError: 184763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5727.207190] Lustre: server umount lustre-MDT0001 complete [ 5739.039413] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 5742.071936] Key type lgssc unregistered [ 5742.398536] LNet: 188348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5742.417749] LNetError: 188348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5742.430785] LNet: Removed LNI 192.168.202.118@tcp [ 5743.396217] Key type .llcrypt unregistered [ 5743.399704] Key type ._llcrypt unregistered [ 5769.675903] Key type ._llcrypt registered [ 5769.677305] Key type .llcrypt registered [ 5769.748706] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 5771.212175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5771.238117] alg: No test for adler32 (adler32-zlib) [ 5772.286885] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 5772.458289] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 5774.064184] Key type lgssc registered [ 5775.561752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5783.866478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5783.894952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5785.429642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5789.904656] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5795.810041] LustreError: 189769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5795.830097] LustreError: 189769:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5799.284643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5799.877657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5804.613846] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5813.682543] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5820.597809] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5829.109884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5829.544156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5830.582020] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5836.017544] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5843.204510] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5850.088682] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5850.097774] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5850.119021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5850.123488] Lustre: Skipped 1 previous similar message [ 5855.204574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5855.215166] Lustre: Skipped 2 previous similar messages [ 5856.087093] Lustre: server umount lustre-OST0000 complete [ 5863.933071] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9f9c50ac4000.ost_server_uuid 50 [ 5865.605048] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5880.806269] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5880.825024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5880.829141] Lustre: Skipped 2 previous similar messages [ 5880.839795] Lustre: Skipped 2 previous similar messages [ 5886.465370] Lustre: server umount lustre-MDT0000 complete [ 5886.945658] LustreError: 189769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5886.961473] LustreError: 189769:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5890.403485] LustreError: 191614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378746 with bad export cookie 6228164146618241608 [ 5890.404481] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5890.409354] LustreError: 191614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5890.639440] Lustre: server umount lustre-MDT0001 complete [ 5899.150469] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:05:53 (1787378753) [ 5905.131527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5905.467979] LustreError: 193684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5905.529452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5909.568639] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5911.013396] LustreError: 193685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5916.130798] LustreError: 193684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5917.151989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5917.497615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5922.592806] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5930.763447] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5936.877593] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5944.768892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5945.041905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5951.241513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5951.707220] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5959.086742] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5965.541801] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5965.707287] Lustre: lustre-OST0001: new disk, initializing [ 5965.710693] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5965.715451] Lustre: 196527:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5965.799687] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5972.563941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5972.573906] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5972.682642] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5972.993640] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5982.409707] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5987.299394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5987.305838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5987.316060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5987.320797] Lustre: Skipped 2 previous similar messages [ 5991.482436] Lustre: server umount lustre-OST0000 complete [ 5993.288758] LustreError: 195835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5997.670449] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5997.840583] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6002.613347] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6002.796786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6013.769854] LustreError: 195835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6013.795606] LustreError: 195835:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6019.049455] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6019.050934] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6019.059265] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6019.059274] Lustre: Skipped 3 previous similar messages [ 6019.066815] Lustre: Skipped 2 previous similar messages [ 6019.091606] LustreError: Skipped 1 previous similar message [ 6023.841404] Lustre: server umount lustre-OST0001 complete [ 6029.291039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6029.297250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6029.309427] Lustre: Skipped 1 previous similar message [ 6029.328068] Lustre: Skipped 2 previous similar messages [ 6034.456851] Lustre: server umount lustre-MDT0000 complete [ 6038.578963] LustreError: 193666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787378894 with bad export cookie 6228164146618242679 [ 6038.586789] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6038.593459] LustreError: 193666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6038.996437] Lustre: server umount lustre-MDT0001 complete [ 6062.951173] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:08:36 (1787378916) [ 6070.135797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6070.477885] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6070.545232] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6070.817749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6075.558350] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6085.265758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6085.383487] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6085.411650] Lustre: 199962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6090.867497] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6100.053716] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6107.158253] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6115.020481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6115.204190] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6121.047667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6122.015592] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6129.309751] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6137.035879] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6137.119426] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6137.369852] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6137.375160] Lustre: Skipped 2 previous similar messages [ 6143.541908] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6152.429892] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6156.770183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6156.775660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6156.790799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6156.796368] Lustre: Skipped 1 previous similar message [ 6161.902661] Lustre: server umount lustre-OST0000 complete [ 6163.804052] LustreError: 201812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6163.844488] LustreError: 201812:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6179.308791] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6179.311428] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6179.317730] LustreError: Skipped 2 previous similar messages [ 6179.345342] Lustre: Skipped 2 previous similar messages [ 6184.687165] Lustre: server umount lustre-OST0001 complete [ 6189.539841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6189.553231] Lustre: Skipped 8 previous similar messages [ 6194.913177] Lustre: server umount lustre-MDT0000 complete [ 6198.505310] LustreError: 199953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379054 with bad export cookie 6228164146618244086 [ 6198.509512] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6198.515410] LustreError: 199953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6198.818759] Lustre: server umount lustre-MDT0001 complete [ 6223.847686] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:11:17 (1787379077) [ 6239.416755] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 6251.696717] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 6293.350141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6293.560824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6293.588157] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6293.645699] Lustre: lustre-MDT0000: new disk, initializing [ 6293.702482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6293.722609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6298.463374] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6309.119008] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6309.278835] Lustre: 208061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6309.342897] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6309.346996] Lustre: Skipped 1 previous similar message [ 6309.405560] Lustre: lustre-MDT0001: new disk, initializing [ 6309.512477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6309.517862] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6314.225291] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6323.550829] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6329.194718] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6337.340156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6337.620386] Lustre: lustre-OST0000: new disk, initializing [ 6337.630435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6337.641714] Lustre: 210029:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6343.260879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6343.273671] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6343.387953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6344.320178] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6355.854158] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6360.638601] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6360.797752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6365.532824] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6365.797305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6369.249533] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6369.258514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6369.258524] Lustre: Skipped 2 previous similar messages [ 6369.259266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6369.259270] Lustre: Skipped 2 previous similar messages [ 6369.307627] LustreError: Skipped 1 previous similar message [ 6374.064449] Lustre: server umount lustre-OST0000 complete [ 6384.615709] LustreError: 208070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6384.628978] LustreError: 208070:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6384.659798] Lustre: server umount lustre-MDT0000 complete [ 6388.517311] LustreError: 210028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379244 with bad export cookie 6228164146618245647 [ 6388.520726] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6388.527281] LustreError: 210028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6388.728390] Lustre: server umount lustre-MDT0001 complete [ 6399.027315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6404.972184] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6413.464842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6418.385789] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6426.096557] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6432.319777] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6439.035877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6445.456962] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6453.129443] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6457.933541] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6458.186371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6464.285075] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6464.491609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6469.090868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6469.099834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6469.110378] Lustre: Skipped 3 previous similar messages [ 6472.812167] Lustre: server umount lustre-OST0000 complete [ 6479.127127] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6483.206619] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6512.741498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6519.796120] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6527.788418] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6529.440604] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6533.447509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6533.484678] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6533.484936] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6533.511801] Lustre: Skipped 1 previous similar message [ 6538.751708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6538.770520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6538.785366] Lustre: Skipped 6 previous similar messages [ 6539.756369] Lustre: server umount lustre-OST0000 complete [ 6549.533149] Lustre: server umount lustre-MDT0000 complete [ 6552.855138] LustreError: 212226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379408 with bad export cookie 6228164146618246522 [ 6552.857185] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6552.868963] LustreError: 212226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6553.080505] Lustre: server umount lustre-MDT0001 complete [ 6561.431865] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:16:55 (1787379415) [ 6567.599478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6567.900494] LustreError: 218208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6567.914773] LustreError: 218208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6567.952757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6567.960240] Lustre: Skipped 6 previous similar messages [ 6571.923749] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6579.735302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6584.547224] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6592.713868] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6599.018183] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6604.842662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6614.765260] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6623.076103] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6628.170925] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6628.612478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6634.184779] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6634.537297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6642.726926] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6642.988962] Lustre: lustre-OST0001: new disk, initializing [ 6642.996174] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6643.007105] Lustre: 221579:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6651.045260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6651.060696] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6651.175073] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6651.204643] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6661.958564] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6668.658715] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6668.888556] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6673.524071] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6673.710388] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6676.961129] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6676.966267] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6676.974226] LustreError: Skipped 1 previous similar message [ 6676.994627] Lustre: Skipped 5 previous similar messages [ 6682.099313] Lustre: server umount lustre-OST0001 complete [ 6691.129420] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6691.534285] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6698.090099] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6698.389148] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6728.851082] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6734.326333] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6734.340624] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6734.344217] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6735.975913] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6744.620032] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6746.504614] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6755.135702] Lustre: server umount lustre-OST0001 complete [ 6778.781901] Lustre: server umount lustre-OST0000 complete [ 6788.960836] Lustre: server umount lustre-MDT0000 complete [ 6792.485381] LustreError: 218188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379648 with bad export cookie 6228164146618247495 [ 6792.496204] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6799.192501] Lustre: server umount lustre-MDT0001 complete [ 6826.006297] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 02:21:19 (1787379679) [ 6833.100251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6833.463693] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6833.526166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6839.027948] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6848.180448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6848.289084] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6848.324069] Lustre: 226826:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6853.275727] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6862.441972] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6868.499411] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6875.790329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6876.051930] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6882.077284] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6889.056458] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6898.293000] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6898.357945] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6904.897852] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6913.226777] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6915.207241] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6920.713606] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6920.957684] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6925.729606] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6925.950519] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6931.971537] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f9c51515000.ost_server_uuid 50 [ 6933.843259] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f9c51515000.ost_server_uuid in IDLE state after 0 sec [ 6934.822875] Lustre: Permanently deactivating lustre-OST0001 [ 6934.861159] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6937.797688] Lustre: Permanently reactivating lustre-OST0001 [ 6946.272998] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6946.285458] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6946.297491] LustreError: Skipped 5 previous similar messages [ 6946.302218] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6946.310568] Lustre: Skipped 7 previous similar messages [ 6946.326543] Lustre: Skipped 17 previous similar messages [ 6948.607554] Lustre: server umount lustre-OST0001 complete [ 6951.397711] LustreError: 228672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6951.410898] LustreError: 228672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 39 previous similar messages [ 6958.577384] Lustre: server umount lustre-OST0000 complete [ 6969.034984] Lustre: server umount lustre-MDT0000 complete [ 6972.908843] LustreError: 226820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379828 with bad export cookie 6228164146618248783 [ 6972.913958] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6972.920139] LustreError: 226820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6973.314424] Lustre: server umount lustre-MDT0001 complete [ 6997.539569] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 02:24:11 (1787379851) [ 7005.781354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7006.013841] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7006.029440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7006.035933] Lustre: Skipped 5 previous similar messages [ 7011.669785] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7022.619942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7022.743889] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7022.785830] Lustre: 233518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7027.788123] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7036.901053] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7044.799090] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7055.150238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7055.326900] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7055.348065] Lustre: Permanently deactivating lustre-OST0000 [ 7055.371736] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7063.583399] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7064.627495] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7072.914961] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7080.902409] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7081.043659] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7081.296918] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7081.300719] Lustre: Skipped 11 previous similar messages [ 7088.479111] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7089.965654] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7097.365307] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7099.232530] Lustre: Permanently reactivating lustre-OST0000 [ 7099.250437] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7099.254250] Lustre: Skipped 2 previous similar messages [ 7100.062273] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7100.066155] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 35 seconds. I think it's dead, and I am evicting it. exp 00000000f88856b9, cur 1787379955 deadline 1787379950 last 1787379920 [ 7100.090159] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7100.101395] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7100.104137] Lustre: Skipped 1 previous similar message [ 7100.147653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7113.270802] Lustre: server umount lustre-OST0001 complete [ 7123.846660] Lustre: server umount lustre-OST0000 complete [ 7134.821738] Lustre: server umount lustre-MDT0000 complete [ 7139.825705] LustreError: 233509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787379995 with bad export cookie 6228164146618250428 [ 7139.847669] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7139.852486] LustreError: 233509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7140.435314] Lustre: server umount lustre-MDT0001 complete [ 7154.479512] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 7157.686941] Key type lgssc unregistered [ 7158.058909] LNet: 238402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7158.072568] LNetError: 238402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7158.094642] LNet: Removed LNI 192.168.202.118@tcp [ 7159.131275] Key type .llcrypt unregistered [ 7159.132795] Key type ._llcrypt unregistered [ 7187.455201] Key type ._llcrypt registered [ 7187.457336] Key type .llcrypt registered [ 7187.619549] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 7189.430048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7189.476130] alg: No test for adler32 (adler32-zlib) [ 7190.689517] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 7191.091075] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 7192.896296] Key type lgssc registered [ 7193.953909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7205.853837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7205.881745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7207.584533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7214.027150] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7218.161235] LustreError: 239870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7218.191421] LustreError: 239870:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7223.279285] LustreError: 239871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7224.269937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7224.520941] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7224.582304] Lustre: 239859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7224.598067] Lustre: Permanently deactivating lustre-MDT0001 [ 7224.614129] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7225.167896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7230.328214] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7231.503652] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7237.697511] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7244.173654] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7253.868054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7254.500352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7261.996272] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7269.874515] LustreError: 241713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7269.905557] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7270.811416] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7274.995900] LustreError: 241713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7278.075667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7278.510461] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7278.810828] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7284.227793] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7284.241047] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7286.088132] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7295.578357] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7299.939509] Lustre: Permanently reactivating lustre-MDT0001 [ 7299.965836] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7299.976390] Lustre: Skipped 2 previous similar messages [ 7301.116336] Lustre: lustre-MDT0001: Client 29342537-6bf7-4a3c-bb40-7854274c9913 (at 192.168.202.18@tcp) reconnecting [ 7307.884617] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7307.906566] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7307.921737] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7307.944367] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7308.574526] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7310.433206] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7317.401485] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7317.694844] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7322.727048] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7323.177945] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7326.352734] Lustre: Permanently deactivating lustre-MDT0001 [ 7326.386909] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7326.396447] Lustre: Skipped 2 previous similar messages [ 7334.599604] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7343.079439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7343.087504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7343.101198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7343.616243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7343.622260] Lustre: Skipped 2 previous similar messages [ 7348.712810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7348.725607] Lustre: Skipped 3 previous similar messages [ 7348.947718] Lustre: server umount lustre-MDT0000 complete [ 7353.220014] LustreError: 239851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380208 with bad export cookie 574741407872165933 [ 7353.221778] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7353.243143] LustreError: 239851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7353.679287] Lustre: server umount lustre-MDT0001 complete [ 7359.024383] Lustre: server umount lustre-OST0000 complete [ 7374.189049] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 02:30:27 (1787380227) [ 7386.361694] Lustre: server umount lustre-OST0001 complete [ 7394.919389] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 7406.186327] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 7448.568591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7448.871279] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7448.880830] Lustre: Skipped 2 previous similar messages [ 7448.912354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7449.009466] Lustre: lustre-MDT0000: new disk, initializing [ 7449.109662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7449.141803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7454.488988] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7465.298588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7465.408950] Lustre: 248132:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7465.457806] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7465.461312] Lustre: Skipped 1 previous similar message [ 7465.526718] Lustre: lustre-MDT0001: new disk, initializing [ 7465.600454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7465.639764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7465.653177] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7469.543511] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7478.287761] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7484.298972] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7492.412855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7492.837524] Lustre: lustre-OST0000: new disk, initializing [ 7492.841347] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7492.847782] Lustre: 250101:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7492.940300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7498.833165] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7502.460751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7502.478043] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7502.602614] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7511.335948] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7513.850113] LustreError: 249395:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7523.835467] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7523.952512] LustreError: 249395:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7524.001095] LustreError: 248138:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7524.081679] Lustre: lustre-OST0001: new disk, initializing [ 7524.085792] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7524.118072] Lustre: 251214:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7532.078841] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7532.264527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7532.287810] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7532.536447] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7534.096433] LustreError: 248138:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7542.123044] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7544.364749] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7547.874453] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7547.878568] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7547.890125] LustreError: Skipped 1 previous similar message [ 7547.902496] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7547.904945] Lustre: Skipped 4 previous similar messages [ 7547.924712] Lustre: Skipped 1 previous similar message [ 7552.571916] Lustre: server umount lustre-OST0001 complete [ 7552.996425] LustreError: 250086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7553.023531] LustreError: 250086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7563.241662] LustreError: 250086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7563.254184] LustreError: 250086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7573.490540] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7573.508669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7573.515491] Lustre: Skipped 1 previous similar message [ 7583.717445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7583.720045] LustreError: 250085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7583.726187] Lustre: Skipped 3 previous similar messages [ 7583.752754] LustreError: 250085:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 7584.224578] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7584.516718] Lustre: server umount lustre-OST0000 complete [ 7593.956749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7593.961398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7593.981734] Lustre: Skipped 1 previous similar message [ 7596.418764] Lustre: server umount lustre-MDT0000 complete [ 7600.778556] LustreError: 248124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380456 with bad export cookie 574741407872168747 [ 7600.781768] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7600.796213] LustreError: 248124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7607.381647] Lustre: server umount lustre-MDT0001 complete [ 7619.594847] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 7623.256850] Key type lgssc unregistered [ 7623.570173] LNet: 253252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7623.580064] LNetError: 253252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7623.627043] LNet: Removed LNI 192.168.202.118@tcp [ 7625.114663] Key type .llcrypt unregistered [ 7625.118949] Key type ._llcrypt unregistered [ 7670.739440] Key type ._llcrypt registered [ 7670.742499] Key type .llcrypt registered [ 7671.008189] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 7672.906894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7672.967449] alg: No test for adler32 (adler32-zlib) [ 7673.985836] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 7674.247119] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 7675.888779] Key type lgssc registered [ 7677.413611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7685.602715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7685.654669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7687.021079] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7687.056453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7687.268366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7692.358564] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7701.643918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7701.707181] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7701.748636] Lustre: 255464:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7702.162678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7707.914459] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7717.989742] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7725.098934] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7733.473401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7733.767234] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7734.142090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7741.188501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7741.233968] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7751.160553] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7807.978657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7807.979702] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7807.985350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7807.992029] Lustre: Skipped 1 previous similar message [ 7808.008893] Lustre: Skipped 1 previous similar message [ 7812.999233] Lustre: server umount lustre-OST0000 complete [ 7818.600921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7831.816264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7832.282128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7839.613747] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7847.683148] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7847.710722] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7847.732203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7847.799392] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7892.840248] Lustre: server umount lustre-OST0000 complete [ 7899.115350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7899.144585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7904.226310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7904.228800] Lustre: Skipped 1 previous similar message [ 7907.532564] Lustre: server umount lustre-MDT0000 complete [ 7909.280635] Lustre: 254787:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787380749/real 1787380749] req@000000005cb35854 x1874204325652224/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787380765 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7909.287401] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7909.354462] Lustre: 254787:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7909.370717] Lustre: Skipped 1 previous similar message [ 7912.384960] LustreError: 257330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380768 with bad export cookie 3604242931382158910 [ 7912.387145] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7912.400812] LustreError: 257330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7912.806641] Lustre: server umount lustre-MDT0001 complete [ 7932.410517] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 7937.630673] Key type lgssc unregistered [ 7938.044660] LNet: 260916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7938.055312] LNetError: 260916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7938.085698] LNet: Removed LNI 192.168.202.118@tcp [ 7939.521054] Key type .llcrypt unregistered [ 7939.527389] Key type ._llcrypt unregistered [ 7970.636996] Key type ._llcrypt registered [ 7970.639792] Key type .llcrypt registered [ 7970.751590] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 7971.914709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7972.028414] alg: No test for adler32 (adler32-zlib) [ 7973.113261] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 7973.410349] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 7975.192352] Key type lgssc registered [ 7976.793651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7987.503045] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7987.521315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7989.288467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7995.962570] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7998.439361] LustreError: 262352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7998.473575] LustreError: 262352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8003.554180] LustreError: 262353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8006.629615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8007.408728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8013.336383] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8022.852643] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8030.173441] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8039.036947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8039.521198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8047.672568] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8059.021715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8060.191552] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8088.045212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8088.061532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8088.079468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8090.081040] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8090.083550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8090.094355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8093.588660] Lustre: server umount lustre-OST0000 complete [ 8099.298926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8099.306703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8099.319065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8099.325432] Lustre: Skipped 1 previous similar message [ 8105.140128] Lustre: server umount lustre-MDT0000 complete [ 8105.447634] LustreError: 262353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8105.468685] LustreError: 262353:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8109.868183] LustreError: 262334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787380965 with bad export cookie 2946637634418060489 [ 8109.879357] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8109.883428] LustreError: 262334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8110.556332] Lustre: server umount lustre-MDT0001 complete [ 8124.335368] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 8128.080821] Key type lgssc unregistered [ 8128.554561] LNet: 266667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8128.566255] LNetError: 266667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8128.584769] LNet: Removed LNI 192.168.202.118@tcp [ 8129.592753] Key type .llcrypt unregistered [ 8129.595844] Key type ._llcrypt unregistered [ 8152.399332] Key type ._llcrypt registered [ 8152.400699] Key type .llcrypt registered [ 8152.498292] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 8153.591650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8153.669385] alg: No test for adler32 (adler32-zlib) [ 8155.091339] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 8155.401041] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 8157.104220] Key type lgssc registered [ 8158.763263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8168.291977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8168.358942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8169.869034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8174.851958] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8180.203150] LustreError: 267891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8180.232563] LustreError: 267891:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8183.541276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8183.918784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8188.328688] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8199.433208] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8207.972934] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8217.133632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8217.438288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8225.156264] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8234.824488] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8240.010792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8250.348097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8250.359509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8250.367537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8255.461716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8255.474552] Lustre: Skipped 1 previous similar message [ 8256.607124] Lustre: server umount lustre-OST0000 complete [ 8265.703296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8265.707036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8265.723476] Lustre: Skipped 1 previous similar message [ 8265.734573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8265.743334] Lustre: Skipped 2 previous similar messages [ 8267.478250] Lustre: server umount lustre-MDT0000 complete [ 8270.822915] LustreError: 267885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8270.845156] LustreError: 267885:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8272.081319] LustreError: 267872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381127 with bad export cookie 3494314350164054423 [ 8272.092073] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8272.101055] LustreError: 267872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8272.537464] Lustre: server umount lustre-MDT0001 complete [ 8282.989477] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 8285.685101] Key type lgssc unregistered [ 8286.001897] LNet: 271574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8286.011568] LNetError: 271574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8286.025895] LNet: Removed LNI 192.168.202.118@tcp [ 8286.997832] Key type .llcrypt unregistered [ 8287.000789] Key type ._llcrypt unregistered [ 8315.087325] Key type ._llcrypt registered [ 8315.089783] Key type .llcrypt registered [ 8315.226868] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 8316.885862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8316.944754] alg: No test for adler32 (adler32-zlib) [ 8318.026798] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 8318.309973] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 8320.072832] Key type lgssc registered [ 8321.578605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8331.495532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8331.528351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8333.027232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8338.532818] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8342.500771] LustreError: 272992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8342.546667] LustreError: 272992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8347.621768] LustreError: 272991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8350.258631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8350.842213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8357.622535] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8367.088892] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8375.618686] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8384.306779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8384.845798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8391.542899] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8400.400907] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8402.330217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8433.121369] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8433.132535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8433.147022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8436.195246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8436.203463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8439.172105] Lustre: server umount lustre-OST0000 complete [ 8446.434129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8446.460101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8446.471384] Lustre: Skipped 2 previous similar messages [ 8449.604926] Lustre: server umount lustre-MDT0000 complete [ 8451.552996] LustreError: 272992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8451.559757] LustreError: 272992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8454.605962] LustreError: 272973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381310 with bad export cookie 13004397886322026960 [ 8454.616991] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8454.621720] LustreError: 272973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8455.307956] Lustre: server umount lustre-MDT0001 complete [ 8469.485889] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 8473.271484] Key type lgssc unregistered [ 8473.620699] LNet: 277308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8473.627221] LNetError: 277308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8473.641260] LNet: Removed LNI 192.168.202.118@tcp [ 8474.830249] Key type .llcrypt unregistered [ 8474.835089] Key type ._llcrypt unregistered [ 8496.914906] Key type ._llcrypt registered [ 8496.922682] Key type .llcrypt registered [ 8497.146855] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 8498.445119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8498.464507] alg: No test for adler32 (adler32-zlib) [ 8499.653255] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 8499.847047] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 8501.554112] Key type lgssc registered [ 8502.903352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8512.671545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8512.709226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8514.601992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8520.472714] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8524.770845] LustreError: 278545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8524.820838] LustreError: 278545:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8529.915510] LustreError: 278547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8531.741334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8532.274241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8537.691271] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8546.414270] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8553.484752] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8562.306121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8562.862546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8569.519505] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8579.062980] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8584.374377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8597.986455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8598.004205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8598.032067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8600.033439] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8600.044585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8600.046822] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8600.090033] Lustre: Skipped 1 previous similar message [ 8603.654694] Lustre: server umount lustre-OST0000 complete [ 8609.257127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8609.272325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8609.300292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8614.366453] Lustre: server umount lustre-MDT0000 complete [ 8615.396066] LustreError: 278554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8615.414700] LustreError: 278554:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8618.559950] LustreError: 278512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381474 with bad export cookie 7733527277232828742 [ 8618.574211] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8618.580753] LustreError: 278512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8618.978842] Lustre: server umount lustre-MDT0001 complete [ 8633.373463] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 8636.149206] Key type lgssc unregistered [ 8636.507767] LNet: 282252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8636.518292] LNetError: 282252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8636.551429] LNet: Removed LNI 192.168.202.118@tcp [ 8637.771064] Key type .llcrypt unregistered [ 8637.774074] Key type ._llcrypt unregistered [ 8651.497920] Key type ._llcrypt registered [ 8651.499849] Key type .llcrypt registered [ 8651.623380] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8673.886461] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 8691.420750] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 8692.315047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8692.333713] alg: No test for adler32 (adler32-zlib) [ 8693.460867] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 8693.700434] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 8695.420422] Key type lgssc registered [ 8696.740500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8735.157512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8735.198612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8736.410127] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8736.433615] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8736.502090] Lustre: lustre-MDT0000: new disk, initializing [ 8736.570578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8736.586991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8741.191955] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8754.797637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8754.982323] Lustre: 285807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8755.035987] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8755.039374] Lustre: Skipped 1 previous similar message [ 8755.100901] Lustre: lustre-MDT0001: new disk, initializing [ 8755.196457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8755.231260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8755.245676] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8759.810459] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8770.372243] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8776.926165] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8784.313511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8784.736418] Lustre: lustre-OST0000: new disk, initializing [ 8784.749355] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8784.762938] Lustre: 287775:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8784.886608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8792.728683] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8793.721435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8793.746827] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8793.918387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8804.063799] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8809.158333] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8809.453215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8814.073302] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8814.337530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8819.684283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8819.685614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8819.692229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8819.706254] Lustre: Skipped 1 previous similar message [ 8822.986864] Lustre: server umount lustre-OST0000 complete [ 8829.931515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8829.943745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8829.953171] Lustre: Skipped 1 previous similar message [ 8829.963375] Lustre: Skipped 2 previous similar messages [ 8833.567563] Lustre: server umount lustre-MDT0000 complete [ 8837.671375] LustreError: 285797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381693 with bad export cookie 11497523703806612965 [ 8837.680565] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8837.688085] LustreError: 285797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8838.028676] Lustre: server umount lustre-MDT0001 complete [ 8847.689993] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 02:55:01 (1787381701) [ 8856.084136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8856.700174] LustreError: 290190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8856.719822] LustreError: 290190:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8856.856883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8862.184257] LustreError: 290191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8863.662699] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8866.274255] LustreError: 290190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8871.406255] LustreError: 290191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8874.313760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8874.860860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8880.266532] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8889.682203] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8897.415035] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8907.520548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8907.989952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8916.341647] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8923.865630] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8941.028089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8941.035147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8941.054308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8942.054223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8944.686468] Lustre: server umount lustre-OST0000 complete [ 8951.783303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8951.792760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8951.814668] Lustre: Skipped 1 previous similar message [ 8951.825796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8951.829603] Lustre: Skipped 2 previous similar messages [ 8954.167831] Lustre: server umount lustre-MDT0000 complete [ 8957.420540] LustreError: 290190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8957.432713] LustreError: 290190:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8958.060928] LustreError: 290171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787381813 with bad export cookie 11497523703806613847 [ 8958.072051] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8958.074586] LustreError: 290171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8958.412564] Lustre: server umount lustre-MDT0001 complete [ 8970.193318] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 8972.886210] Key type lgssc unregistered [ 8973.188678] LNet: 293755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8973.201499] LNetError: 293755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8973.216621] LNet: Removed LNI 192.168.202.118@tcp [ 8974.070760] Key type .llcrypt unregistered [ 8974.072494] Key type ._llcrypt unregistered [ 9005.914849] Key type ._llcrypt registered [ 9005.916494] Key type .llcrypt registered [ 9006.030418] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 9023.352411] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 9024.419544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9024.498288] alg: No test for adler32 (adler32-zlib) [ 9025.581227] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 9025.964262] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 9027.657249] Key type lgssc registered [ 9028.943345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9076.068610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9076.096459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9077.312564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9077.333060] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9077.402994] Lustre: lustre-MDT0000: new disk, initializing [ 9077.540149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9077.553735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9081.534845] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9091.467609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9091.551687] Lustre: 297582:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9091.599386] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9091.602449] Lustre: Skipped 1 previous similar message [ 9091.667706] Lustre: lustre-MDT0001: new disk, initializing [ 9091.787194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9091.831110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9091.840155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9096.217383] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9105.727344] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9111.787578] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9119.939685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9120.189327] Lustre: lustre-OST2710: new disk, initializing [ 9120.192624] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9120.197822] Lustre: 299552:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9120.282787] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9127.604815] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9128.008741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9128.013310] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9128.096453] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9137.813714] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9144.360213] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9144.509771] Lustre: lustre-OST03e8: new disk, initializing [ 9144.522860] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9144.528324] Lustre: 300616:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9144.608806] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9150.653871] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9151.023619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9151.028808] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9151.079294] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9158.332726] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9165.759901] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9166.007183] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9170.848832] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9171.004496] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9175.345610] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9175.541755] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9180.262908] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9180.383621] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9186.787787] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9186.810082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9186.810647] Lustre: Skipped 1 previous similar message [ 9187.777186] Lustre: server umount lustre-MDT0000 complete [ 9191.367153] LustreError: 297577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382047 with bad export cookie 16285631271975700609 [ 9191.377901] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9191.384764] LustreError: 297577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9191.713943] Lustre: server umount lustre-MDT0001 complete [ 9205.969407] Lustre: server umount lustre-OST2710 complete [ 9207.786151] Lustre: 295383:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787382047/real 1787382047] req@00000000da1140ca x1874205743319424/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787382063 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9207.832759] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9207.859060] Lustre: Skipped 2 previous similar messages [ 9210.943723] Lustre: server umount lustre-OST03e8 complete [ 9229.588497] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 9239.940327] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 9277.514896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9277.769713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9277.794937] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9277.868631] Lustre: lustre-MDT0000: new disk, initializing [ 9277.933529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9277.946268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9282.896261] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9293.555737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9293.711733] Lustre: 305881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9293.776686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9293.779947] Lustre: Skipped 1 previous similar message [ 9293.845350] Lustre: lustre-MDT0001: new disk, initializing [ 9293.945775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9293.987394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9294.001615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9298.266934] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9309.655460] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9315.248307] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9323.592290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9323.990864] Lustre: lustre-OST0000: new disk, initializing [ 9323.996665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9324.004433] Lustre: 307848:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9324.114979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9330.674809] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9330.733153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9330.743451] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9330.817199] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9339.758437] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9344.804468] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9345.013704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9350.234408] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9350.429591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9356.259269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9356.264096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9356.270489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9356.291179] Lustre: Skipped 4 previous similar messages [ 9358.496855] Lustre: server umount lustre-OST0000 complete [ 9365.473870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9365.478943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9365.489049] Lustre: Skipped 1 previous similar message [ 9365.496301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9369.327217] Lustre: server umount lustre-MDT0000 complete [ 9372.866542] LustreError: 305874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382228 with bad export cookie 16285631271975702289 [ 9372.882140] LustreError: 305874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9372.882511] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9373.131834] Lustre: server umount lustre-MDT0001 complete [ 9381.189700] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:03:55 (1787382235) [ 9382.612217] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9384.162493] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:03:58 (1787382238) [ 9388.865638] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 9422.511909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9422.777680] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9422.826557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9423.009239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9428.644224] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9438.003182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9438.138626] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9438.225647] Lustre: 311708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9443.597656] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9466.386071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9466.628933] Lustre: Denying initial registration attempt from nid 192.168.202.118@tcp, specified as failover [ 9466.634651] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9466.640462] LustreError: 313162:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9466.645488] LustreError: 313162:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9466.649050] LustreError: 313162:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9466.727560] Lustre: server umount lustre-OST0000 complete [ 9466.729660] LustreError: 313162:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9474.540637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9474.545200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9474.567564] Lustre: Skipped 1 previous similar message [ 9474.581523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9474.584874] Lustre: Skipped 2 previous similar messages [ 9481.185776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9481.189074] Lustre: Skipped 3 previous similar messages [ 9487.328905] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9487.447790] LustreError: 313531:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (00000000e8258fef) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 9489.541274] Lustre: server umount lustre-MDT0000 complete [ 9491.426376] LustreError: 313164:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9491.437152] LustreError: 313164:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9493.156636] LustreError: 311703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382348 with bad export cookie 16285631271975703164 [ 9493.165803] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9493.171454] LustreError: 311703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9493.509853] Lustre: server umount lustre-MDT0001 complete [ 9505.133500] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 9507.923694] Key type lgssc unregistered [ 9508.232741] LNet: 314287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9508.236995] LNetError: 314287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9508.247918] LNet: Removed LNI 192.168.202.118@tcp [ 9509.055798] Key type .llcrypt unregistered [ 9509.058840] Key type ._llcrypt unregistered [ 9520.904095] Key type ._llcrypt registered [ 9520.905885] Key type .llcrypt registered [ 9521.024789] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 9521.869105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9521.913090] alg: No test for adler32 (adler32-zlib) [ 9522.946156] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 9523.160620] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 9524.864192] Key type lgssc registered [ 9525.922827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9571.260438] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 9579.749845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9579.771476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9580.979529] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9581.002922] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9581.132219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9585.958739] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9594.910558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9595.001773] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9595.030698] Lustre: 317052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9595.322917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9599.399408] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9607.163224] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9613.494385] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9633.143676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9633.409101] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9633.651657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9640.635304] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9649.694172] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9652.194804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9652.200740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9652.212381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9652.709874] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9652.713548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9652.749830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9657.825760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9657.830542] Lustre: Skipped 1 previous similar message [ 9658.219312] Lustre: server umount lustre-OST0000 complete [ 9662.946646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9662.956086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9662.965927] Lustre: Skipped 1 previous similar message [ 9662.986179] Lustre: Skipped 1 previous similar message [ 9668.068840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9668.079665] Lustre: Skipped 1 previous similar message [ 9668.887807] Lustre: server umount lustre-MDT0000 complete [ 9673.216657] LustreError: 318920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382528 with bad export cookie 12815833846278079687 [ 9673.223373] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9673.243694] LustreError: 318920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9673.563166] Lustre: server umount lustre-MDT0001 complete [ 9686.720307] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 9689.756477] Key type lgssc unregistered [ 9690.138288] LNet: 320629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9690.143096] LNetError: 320629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9691.175048] LNet: Removed LNI 192.168.202.118@tcp [ 9692.132560] Key type .llcrypt unregistered [ 9692.134397] Key type ._llcrypt unregistered [ 9722.065691] Key type ._llcrypt registered [ 9722.076255] Key type .llcrypt registered [ 9722.224323] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 9723.210422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9723.320918] alg: No test for adler32 (adler32-zlib) [ 9724.448246] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 9724.765981] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 9726.552214] Key type lgssc registered [ 9727.625561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9735.540297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9735.560779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9737.127578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9741.455218] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9746.409755] LustreError: 322051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9746.424153] LustreError: 322051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9749.475270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9749.878649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9754.532537] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9762.828339] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9769.077176] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9777.263985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9777.735105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9783.762606] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9792.109335] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9806.311074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9806.318380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9806.350910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9809.273440] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.18@tcp (stopping) [ 9810.408291] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9810.413636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9810.432147] Lustre: Skipped 1 previous similar message [ 9810.434026] Lustre: Skipped 1 previous similar message [ 9811.955368] Lustre: server umount lustre-MDT0000 complete [ 9814.339443] LustreError: 323291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9815.522935] LustreError: 322051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9815.533187] LustreError: 322051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9815.836672] LustreError: 322031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382671 with bad export cookie 16852984012480578540 [ 9815.845485] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9815.852539] LustreError: 322031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9819.458946] LustreError: 323291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9819.490991] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.18@tcp (stopping) [ 9819.493644] Lustre: Skipped 1 previous similar message [ 9820.671202] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9824.595769] LustreError: 322047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9824.596249] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.18@tcp (stopping) [ 9824.629326] LustreError: 322047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9824.649771] Lustre: Skipped 1 previous similar message [ 9830.368145] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9830.555691] Lustre: server umount lustre-MDT0001 complete [ 9836.937218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9844.713465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9862.105089] LustreError: 323885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9862.126839] LustreError: 323885:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9862.218629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9862.291488] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9867.575578] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9877.947800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9878.374181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9881.457344] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9883.295215] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9891.428772] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9893.870237] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9901.239615] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9903.048591] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9909.218065] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9909.226442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9909.250350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9909.259902] Lustre: Skipped 3 previous similar messages [ 9923.040152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9923.331912] Lustre: server umount lustre-OST0000 complete [ 9927.660109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9927.671368] Lustre: Skipped 1 previous similar message [ 9927.685405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9927.695990] Lustre: Skipped 7 previous similar messages [ 9933.509126] Lustre: server umount lustre-MDT0000 complete [ 9937.802339] LustreError: 322032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382793 with bad export cookie 16852984012480586198 [ 9937.808630] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9937.819402] LustreError: 322032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9938.160696] Lustre: server umount lustre-MDT0001 complete [ 9951.254541] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 9955.346987] Key type lgssc unregistered [ 9955.741524] LNet: 328349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9955.752776] LNetError: 328349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9956.779791] LNet: Removed LNI 192.168.202.118@tcp [ 9958.325871] Key type .llcrypt unregistered [ 9958.328287] Key type ._llcrypt unregistered [ 9986.104984] Key type ._llcrypt registered [ 9986.106368] Key type .llcrypt registered [ 9986.218932] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 9987.506426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9987.572101] alg: No test for adler32 (adler32-zlib) [ 9988.669407] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [ 9989.021448] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 9990.680174] Key type lgssc registered [ 9992.024748] Lustre: Echo OBD driver; http://www.lustre.org/ [10000.386926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10000.412746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10001.838699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10005.692836] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10012.130967] LustreError: 329769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10012.147782] LustreError: 329769:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10013.963143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10014.183800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10018.022621] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10025.341077] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10031.631739] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10036.728595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10036.729494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10036.744197] Lustre: Skipped 1 previous similar message [10036.771022] Lustre: Skipped 1 previous similar message [10038.823292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10038.825974] Lustre: Skipped 1 previous similar message [10041.716905] Lustre: server umount lustre-MDT0000 complete [10043.881815] LustreError: 329765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10043.911370] LustreError: 329765:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10045.521802] LustreError: 329748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787382901 with bad export cookie 13698616400893073349 [10045.528107] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10045.531233] LustreError: 329748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10045.974922] Lustre: server umount lustre-MDT0001 complete [10064.314704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10064.624465] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10064.679838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10065.056538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10070.264689] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10079.571102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10079.675724] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10079.708154] Lustre: 332356:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10080.041805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10085.287553] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10093.437590] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10100.363780] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10111.636272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10111.963675] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10111.988144] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10112.004814] LustreError: 334239:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10112.017737] LustreError: 334239:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10112.024387] LustreError: 334239:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10112.132761] Lustre: server umount lustre-OST0000 complete [10112.138700] LustreError: 334239:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10124.309712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10124.639096] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10124.923626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10131.755409] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10133.738886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10141.434148] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10150.395466] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10150.502068] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10150.606309] Lustre: lustre-OST0001: new disk, initializing [10150.613406] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10150.620427] Lustre: 335784:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10150.720385] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10156.646130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10156.660986] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10156.782075] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10157.610967] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10166.148484] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10172.387052] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10172.387981] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10172.394942] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10172.410974] Lustre: Skipped 1 previous similar message [10175.269093] Lustre: server umount lustre-OST0001 complete [10177.510704] LustreError: 334709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10177.539758] LustreError: 334709:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10182.653742] LustreError: 334710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10182.657156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10182.671126] LustreError: 334710:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10182.676496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10182.682662] Lustre: Skipped 1 previous similar message [10182.694380] Lustre: Skipped 2 previous similar messages [10186.559558] Lustre: server umount lustre-OST0000 complete [10192.876651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10192.882735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10192.903786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10192.913771] Lustre: Skipped 1 previous similar message [10203.108282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10203.111232] Lustre: Skipped 3 previous similar messages [10204.128236] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10204.307234] Lustre: server umount lustre-MDT0000 complete [10207.669647] LustreError: 333096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383063 with bad export cookie 13698616400893073902 [10207.672384] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10207.679437] LustreError: 333096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10207.918306] Lustre: server umount lustre-MDT0001 complete [10217.528042] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [10219.752225] Key type lgssc unregistered [10220.064794] LNet: 337747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10220.070643] LNetError: 337747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10220.086617] LNet: Removed LNI 192.168.202.118@tcp [10220.834086] Key type .llcrypt unregistered [10220.835551] Key type ._llcrypt unregistered [10290.473248] Key type ._llcrypt registered [10290.475301] Key type .llcrypt registered [10290.601195] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [10304.220503] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [10305.576553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10305.605358] alg: No test for adler32 (adler32-zlib) [10306.588852] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [10306.725101] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [10308.345676] Key type lgssc registered [10309.395745] Lustre: Echo OBD driver; http://www.lustre.org/ [10345.932673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10345.974056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10347.190062] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10347.221468] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10347.341141] Lustre: lustre-MDT0000: new disk, initializing [10347.407876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10347.422246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10351.729089] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10361.718750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10361.903914] Lustre: 342685:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10362.029850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10362.036951] Lustre: Skipped 1 previous similar message [10362.141981] Lustre: lustre-MDT0001: new disk, initializing [10362.231744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10362.293561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10362.307786] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10367.040663] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10378.526416] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10385.037691] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10392.715900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10392.988522] Lustre: lustre-OST0000: new disk, initializing [10392.991504] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10392.996258] Lustre: 344652:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10393.084925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10398.802151] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10400.362543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10400.380569] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10400.461395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10407.077571] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10410.836066] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10411.087334] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10415.695319] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10415.887580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10421.222846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10421.224613] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10421.227302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10421.238773] Lustre: Skipped 1 previous similar message [10421.259692] LustreError: Skipped 1 previous similar message [10423.917953] Lustre: server umount lustre-OST0000 complete [10428.901325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10428.908718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10428.935613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10428.941476] Lustre: Skipped 1 previous similar message [10431.458792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10431.459777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10431.476703] Lustre: Skipped 1 previous similar message [10434.414776] Lustre: server umount lustre-MDT0000 complete [10438.468497] LustreError: 342676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383294 with bad export cookie 12809282174202597513 [10438.494533] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10438.506163] LustreError: 342676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10438.918577] Lustre: server umount lustre-MDT0001 complete [10449.246554] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 03:21:43 (1787383303) [10457.058903] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 03:21:51 (1787383311) [10464.777709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10465.205529] LustreError: 347414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10465.224604] LustreError: 347414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10465.287838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10470.073833] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10470.374770] LustreError: 347415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10474.470620] LustreError: 347414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10478.338189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10478.819657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10483.692829] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10492.089995] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10498.638844] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10505.993857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10506.361612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10512.620694] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10520.759255] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10533.562307] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10535.574410] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10537.260654] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10539.130607] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10541.133574] Lustre: Failing over lustre-MDT0000 [10541.487447] Lustre: server umount lustre-MDT0000 complete [10544.480088] LustreError: 347409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10544.621621] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10544.630298] Lustre: Skipped 1 previous similar message [10549.573718] LustreError: 348629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10549.582434] LustreError: 348629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10559.825071] LustreError: 347409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10559.847991] LustreError: 347409:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10560.993153] Lustre: 340908:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787383400/real 1787383400] req@0000000092a2bc35 x1874207086548992/t0(0) o400->MGC192.168.202.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787383416 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10561.035108] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10561.605415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10571.243996] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb1c3b6455d16d815 [10571.268220] Lustre: MGC192.168.202.118@tcp: Connection restored to 0@lo (at 0@lo) [10571.612486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10571.667888] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10575.170419] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10576.881564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10576.963280] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10577.028660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10577.086462] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10587.425977] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10588.960886] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10592.341665] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10594.211310] Lustre: Failing over lustre-MDT0000 [10594.622847] Lustre: server umount lustre-MDT0000 complete [10595.679843] LustreError: 347410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10595.690870] LustreError: 347410:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [10597.349480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10597.350589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10597.367637] Lustre: Skipped 2 previous similar messages [10613.792121] Lustre: 340910:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787383453/real 1787383453] req@000000000bfc60de x1874207086570496/t0(0) o400->MGC192.168.202.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787383469 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10613.823708] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10614.347146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10624.245838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10624.312141] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10626.377583] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10629.183865] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10629.637593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10629.646453] Lustre: Skipped 2 previous similar messages [10629.675695] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10629.732262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10637.901514] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10639.759857] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10644.961679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10644.965774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10644.972300] Lustre: Skipped 2 previous similar messages [10644.983380] Lustre: Skipped 2 previous similar messages [10650.088432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10650.097882] Lustre: Skipped 2 previous similar messages [10651.002909] Lustre: server umount lustre-MDT0000 complete [10654.065391] LustreError: 347396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383509 with bad export cookie 12809282174202600236 [10654.074285] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10654.080797] LustreError: 347396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10654.433272] Lustre: server umount lustre-MDT0001 complete [10667.571170] Lustre: server umount lustre-OST0000 complete [10683.945429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10684.388329] LustreError: 353777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10684.407499] LustreError: 353777:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [10689.310837] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10697.423088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10697.837381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10697.855653] Lustre: Skipped 1 previous similar message [10702.740879] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10711.630506] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10717.611960] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10724.122104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10729.648443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10730.311238] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10737.745683] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10748.400168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10748.409105] Lustre: Skipped 1 previous similar message [10748.419102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10748.427226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10748.433539] Lustre: Skipped 1 previous similar message [10749.484839] Lustre: server umount lustre-OST0000 complete [10754.022320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10768.352480] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10768.673244] Lustre: server umount lustre-MDT0000 complete [10768.872363] LustreError: 354506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10768.897508] LustreError: 354506:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10772.631120] LustreError: 354513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383628 with bad export cookie 12809282174202600978 [10772.645221] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10772.646017] LustreError: 354513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10772.897604] Lustre: server umount lustre-MDT0001 complete [10788.755105] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [10793.693503] Key type lgssc unregistered [10794.084035] LNet: 357317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10794.106416] LNetError: 357317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10795.175271] LNet: Removed LNI 192.168.202.118@tcp [10796.770768] Key type .llcrypt unregistered [10796.778206] Key type ._llcrypt unregistered [10830.250122] Key type ._llcrypt registered [10830.256543] Key type .llcrypt registered [10830.370442] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [10831.438338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10831.469117] alg: No test for adler32 (adler32-zlib) [10832.766734] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [10833.143303] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [10834.824324] Key type lgssc registered [10836.105948] Lustre: Echo OBD driver; http://www.lustre.org/ [10845.046240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10845.115030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10846.820434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10851.897954] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10856.928960] LustreError: 358833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10856.960091] LustreError: 358833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10862.049073] LustreError: 358834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10862.357594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10862.720652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10867.289584] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10875.206554] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10881.515096] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10888.969614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10889.251242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10891.314448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10895.385740] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10903.609759] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10922.977945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10922.988177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10922.993379] Lustre: Skipped 2 previous similar messages [10923.008519] Lustre: Skipped 2 previous similar messages [10925.723443] Lustre: server umount lustre-MDT0000 complete [10928.098988] LustreError: 358833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10928.123887] LustreError: 358833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10929.827136] LustreError: 358816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383785 with bad export cookie 13060490943117333323 [10929.841623] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10929.848111] LustreError: 358816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10930.331533] Lustre: server umount lustre-MDT0001 complete [10944.881476] Lustre: server umount lustre-OST0000 complete [10960.518760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10961.328093] LustreError: 362563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10961.432207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10965.932728] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10966.501254] LustreError: 362564:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10974.185423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10974.542647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10978.450510] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10986.193667] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10991.672069] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10997.860601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10998.182104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11004.055863] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11006.392543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11011.094413] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11025.889851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11025.893669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11025.901666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11026.914250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11026.930201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11029.302626] Lustre: server umount lustre-MDT0000 complete [11032.033337] LustreError: 362563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11032.055073] LustreError: 362563:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11033.667527] LustreError: 362545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383889 with bad export cookie 13060490943117334436 [11033.671216] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11033.675686] LustreError: 362545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11034.094461] Lustre: server umount lustre-MDT0001 complete [11048.730775] Lustre: server umount lustre-OST0000 complete [11067.198210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11067.713523] LustreError: 366473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11067.834535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11073.014640] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11083.995864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11089.121746] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11098.954617] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11106.923390] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11115.118166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11115.530649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11115.534750] Lustre: Skipped 1 previous similar message [11121.335609] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11129.000591] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11138.103329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11148.259552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11148.268723] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11148.284612] Lustre: Skipped 1 previous similar message [11148.289492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11148.301597] Lustre: Skipped 3 previous similar messages [11151.707880] Lustre: server umount lustre-OST0000 complete [11155.937577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11155.947112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11155.980320] Lustre: Skipped 1 previous similar message [11155.998600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11161.699217] Lustre: server umount lustre-MDT0000 complete [11163.619472] LustreError: 367202:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11163.640878] LustreError: 367202:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11166.081961] LustreError: 366453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384021 with bad export cookie 13060490943117335458 [11166.098294] LustreError: 366453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11166.101908] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11166.589620] Lustre: server umount lustre-MDT0001 complete [11178.726930] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [11181.690588] Key type lgssc unregistered [11181.965071] LNet: 370036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11181.971228] LNetError: 370036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11183.016702] LNet: Removed LNI 192.168.202.118@tcp [11183.833660] Key type .llcrypt unregistered [11183.838097] Key type ._llcrypt unregistered [11211.980514] Key type ._llcrypt registered [11211.984063] Key type .llcrypt registered [11212.082413] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [11213.057391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11213.148169] alg: No test for adler32 (adler32-zlib) [11214.169823] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [11214.343563] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [11216.000810] Key type lgssc registered [11217.061994] Lustre: Echo OBD driver; http://www.lustre.org/ [11224.339378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11224.358153] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11224.363349] LustreError: 371475:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11224.377219] LustreError: 371475:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11224.381320] LustreError: 371475:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11224.386084] LustreError: 371475:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11224.391851] LustreError: 371475:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11231.296896] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11231.311199] LustreError: 371872:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11231.331196] LustreError: 371872:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11231.336456] LustreError: 371872:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11231.342382] LustreError: 371872:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11231.348436] LustreError: 371872:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11253.582610] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [11256.104317] Key type lgssc unregistered [11256.474082] LNet: 372879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11256.485107] LNetError: 372879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11256.506478] LNet: Removed LNI 192.168.202.118@tcp [11257.605280] Key type .llcrypt unregistered [11257.607232] Key type ._llcrypt unregistered [11275.926965] Key type ._llcrypt registered [11275.933618] Key type .llcrypt registered [11276.041264] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [11291.563329] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [11292.893621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11293.010719] alg: No test for adler32 (adler32-zlib) [11294.092840] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [11294.321234] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [11295.976177] Key type lgssc registered [11297.084895] Lustre: Echo OBD driver; http://www.lustre.org/ [11335.008552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11335.022769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11336.314239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11336.337393] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11336.389459] Lustre: lustre-MDT0000: new disk, initializing [11336.436861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11336.450811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11341.202891] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11351.833641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11351.943179] Lustre: 376034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11352.006483] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11352.015123] Lustre: Skipped 1 previous similar message [11352.068321] Lustre: lustre-MDT0001: new disk, initializing [11352.115636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11352.134229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11352.143266] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11356.759177] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11368.043392] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11374.022689] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11381.111382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11381.486487] Lustre: lustre-OST0000: new disk, initializing [11381.498516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11381.527722] Lustre: 378006:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11381.632699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11388.752342] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11390.003577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11390.014556] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11390.167124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11398.794973] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11403.700729] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11403.906460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11408.819277] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11409.076726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11416.036072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11416.050259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11416.065953] Lustre: Skipped 1 previous similar message [11416.088831] Lustre: Skipped 1 previous similar message [11417.415346] Lustre: server umount lustre-OST0000 complete [11423.713859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11423.726693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11423.747683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11426.273427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11426.274849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11427.729560] Lustre: server umount lustre-MDT0000 complete [11431.248974] LustreError: 376027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384287 with bad export cookie 17343524041828997043 [11431.260321] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11431.263689] LustreError: 376027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11431.948533] Lustre: server umount lustre-MDT0001 complete [11442.261361] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11444.477094] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 03:38:17 (1787384297) [11453.155087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11453.724702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11459.047666] LustreError: 380467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11459.080816] LustreError: 380467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11459.542532] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11464.169048] LustreError: 380466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11469.288195] LustreError: 380467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11469.929776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11470.246280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11474.813566] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11483.990644] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11490.609943] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11496.912410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11497.151835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11502.727767] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11509.481104] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11515.944245] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11516.131661] Lustre: lustre-OST0001: new disk, initializing [11516.135447] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11516.147389] Lustre: 383308:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11516.250491] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11522.127103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11522.144647] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11522.272724] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11522.744869] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11532.602709] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11537.893992] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11537.912966] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11537.927596] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11537.932716] Lustre: Skipped 1 previous similar message [11537.936688] Lustre: Skipped 1 previous similar message [11542.501528] Lustre: server umount lustre-OST0001 complete [11543.011100] LustreError: 382292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11543.027354] LustreError: 382292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11548.134070] LustreError: 382292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11548.147664] LustreError: 382292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11558.373151] LustreError: 383592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11558.398982] LustreError: 383592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11560.929395] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11560.937717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11560.950421] LustreError: Skipped 2 previous similar messages [11560.956430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11560.956439] Lustre: Skipped 2 previous similar messages [11561.013310] Lustre: Skipped 1 previous similar message [11566.668070] Lustre: server umount lustre-OST0000 complete [11572.706709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11572.712870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11572.726344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11572.732233] Lustre: Skipped 3 previous similar messages [11577.999450] Lustre: server umount lustre-MDT0000 complete [11578.850695] LustreError: 380467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11578.861695] LustreError: 380467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11581.328254] LustreError: 380447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384437 with bad export cookie 17343524041828997925 [11581.336186] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11587.629934] Lustre: server umount lustre-MDT0001 complete [11600.168494] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [11602.558702] Key type lgssc unregistered [11602.840140] LNet: 385299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11602.858527] LNetError: 385299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11602.893347] LNet: Removed LNI 192.168.202.118@tcp [11603.625718] Key type .llcrypt unregistered [11603.628349] Key type ._llcrypt unregistered [11664.773780] Key type ._llcrypt registered [11664.776583] Key type .llcrypt registered [11664.895479] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [11666.354335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11666.400254] alg: No test for adler32 (adler32-zlib) [11667.529912] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [11667.811950] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [11669.520842] Key type lgssc registered [11670.894278] Lustre: Echo OBD driver; http://www.lustre.org/ [11678.736413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11678.751499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11680.015835] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11680.035676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11680.214678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11684.548183] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11692.767914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11692.825800] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11692.858185] Lustre: 388099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11693.111611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11697.498405] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11705.394227] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11712.122720] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11715.556930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11715.562942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11715.582271] Lustre: Skipped 1 previous similar message [11715.595439] Lustre: Skipped 1 previous similar message [11720.796967] Lustre: server umount lustre-MDT0000 complete [11724.538000] LustreError: 388094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384580 with bad export cookie 9173632431762270602 [11724.549279] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11724.553876] LustreError: 388094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11725.178814] Lustre: server umount lustre-MDT0001 complete [11736.009833] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 03:43:09 (1787384589) [11743.226767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11743.642623] LustreError: 390611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11743.655048] LustreError: 390611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11743.724051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11748.842470] LustreError: 390612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11749.685797] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11753.953943] LustreError: 390611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11759.076541] LustreError: 390612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11761.612993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11762.602094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11771.342145] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11781.995448] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11788.987879] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11796.617725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11796.918110] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11797.121266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11804.216984] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11812.557710] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11820.249254] Lustre: Permanently deactivating lustre-OST0000 [11820.264761] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11830.356463] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11831.398872] LustreError: 393649:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11831.406632] LustreError: 393649:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11831.410905] LustreError: 393649:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11834.522240] LustreError: 393698:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11834.527280] LustreError: 393698:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11842.921847] Lustre: server umount lustre-OST0000 complete [11845.773362] LustreError: 393947:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11845.778730] LustreError: 393947:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11849.698741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11849.703307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11849.717718] Lustre: Skipped 1 previous similar message [11849.719487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11849.749815] Lustre: Skipped 1 previous similar message [11854.819167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11854.833948] Lustre: Skipped 1 previous similar message [11857.986537] Lustre: server umount lustre-MDT0000 complete [11859.949040] LustreError: 390612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11859.974679] LustreError: 390612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11861.990579] LustreError: 392447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384717 with bad export cookie 9173632431762271183 [11861.998792] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11862.001268] LustreError: 392447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11872.436500] Lustre: server umount lustre-MDT0001 complete [11882.750198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11888.272497] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11891.654248] LustreError: 395109:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11899.318875] Lustre: 395448:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.118@tcp [11901.840520] Lustre: server umount MGS complete [11915.055160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11915.536945] LustreError: 396173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11915.553443] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11915.564381] Lustre: Skipped 1 previous similar message [11915.605263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11920.854361] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11931.120423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11931.282195] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11936.465079] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11944.943184] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11951.790869] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11954.292574] Lustre: Permanently reactivating lustre-OST0000 [11954.316820] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11954.324629] Lustre: Skipped 2 previous similar messages [11980.768352] LustreError: 397904:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11987.821086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11988.176065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11988.182384] Lustre: Skipped 1 previous similar message [11996.758918] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12008.664371] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12021.220042] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12021.224741] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12021.230550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12021.238663] Lustre: Skipped 1 previous similar message [12024.597979] Lustre: server umount lustre-OST0000 complete [12031.458651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12031.479363] Lustre: Skipped 1 previous similar message [12031.492028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12035.979180] Lustre: server umount lustre-MDT0000 complete [12036.590398] LustreError: 396168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12036.610136] LustreError: 396168:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12041.060165] LustreError: 396155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384896 with bad export cookie 9173632431762272233 [12041.068276] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12041.087927] LustreError: 396155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12041.521293] Lustre: server umount lustre-MDT0001 complete [12057.269628] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [12061.157332] Key type lgssc unregistered [12061.578844] LNet: 400494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12061.598930] LNetError: 400494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12061.617935] LNet: Removed LNI 192.168.202.118@tcp [12062.889200] Key type .llcrypt unregistered [12062.891951] Key type ._llcrypt unregistered [12085.136690] Key type ._llcrypt registered [12085.139605] Key type .llcrypt registered [12085.256721] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [12103.329757] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [12104.379861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12104.548649] alg: No test for adler32 (adler32-zlib) [12105.582334] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [12105.867096] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [12107.528264] Key type lgssc registered [12108.913192] Lustre: Echo OBD driver; http://www.lustre.org/ [12148.315908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12148.340981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12149.716440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12149.764446] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12149.831284] Lustre: lustre-MDT0000: new disk, initializing [12149.878335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12149.910474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12155.457081] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12165.893990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12166.060738] Lustre: 403659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12166.146133] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12166.149055] Lustre: Skipped 1 previous similar message [12166.195119] Lustre: lustre-MDT0001: new disk, initializing [12166.259426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12166.316627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12166.334726] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12171.191772] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12180.714530] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12186.653208] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12193.164440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12193.424671] Lustre: lustre-OST0000: new disk, initializing [12193.430025] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12193.438267] Lustre: 405627:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12193.561281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12198.652753] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12199.992977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12199.998462] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12200.151507] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12208.493471] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12214.437532] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12214.710835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12219.816567] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12219.996905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12226.022635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12226.023174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12226.051653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12231.145236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12231.157291] Lustre: Skipped 2 previous similar messages [12236.258885] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12236.271578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12236.285720] Lustre: Skipped 1 previous similar message [12236.576922] Lustre: server umount lustre-OST0000 complete [12241.379692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12241.391685] Lustre: Skipped 1 previous similar message [12241.404838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12241.410812] Lustre: Skipped 1 previous similar message [12246.497154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12247.001643] Lustre: server umount lustre-MDT0000 complete [12250.819894] LustreError: 403650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385106 with bad export cookie 3473531915397924002 [12250.821133] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12250.829882] LustreError: 403650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12251.065921] Lustre: server umount lustre-MDT0001 complete [12259.463240] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12260.921461] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 03:51:54 (1787385114) [12268.744940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12269.414109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12274.096174] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12274.659289] LustreError: 408087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12274.684918] LustreError: 408087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12279.778118] LustreError: 408088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12282.420486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12282.844178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12288.131368] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12295.579286] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12301.968630] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12309.059782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12309.553690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12315.440804] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12323.675781] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12325.631698] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12328.443822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12335.078556] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12335.083250] LustreError: Skipped 1 previous similar message [12335.090023] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12335.099719] Lustre: Skipped 1 previous similar message [12335.113076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12335.115505] Lustre: Skipped 1 previous similar message [12338.146991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12338.157618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12339.994847] Lustre: server umount lustre-OST0000 complete [12344.292109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12344.298650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12350.681440] Lustre: server umount lustre-MDT0000 complete [12353.513874] LustreError: 408084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12353.559099] LustreError: 408084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12355.051195] LustreError: 408069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385210 with bad export cookie 3473531915397924884 [12355.058459] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12355.065188] LustreError: 408069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12355.572483] Lustre: server umount lustre-MDT0001 complete [12368.673666] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [12371.769438] Key type lgssc unregistered [12372.006355] LNet: 411814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12372.027515] LNetError: 411814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12373.099091] LNet: Removed LNI 192.168.202.118@tcp [12374.280991] Key type .llcrypt unregistered [12374.283909] Key type ._llcrypt unregistered [12402.657406] Key type ._llcrypt registered [12402.659436] Key type .llcrypt registered [12402.795804] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [12403.095578] Key type .llcrypt unregistered [12403.098577] Key type ._llcrypt unregistered [12425.537630] Key type ._llcrypt registered [12425.539739] Key type .llcrypt registered [12425.657774] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [12427.219549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12427.280461] alg: No test for adler32 (adler32-zlib) [12428.542621] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [12428.781694] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [12430.488269] Key type lgssc registered [12432.308295] Lustre: Echo OBD driver; http://www.lustre.org/ [12442.561087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12442.609352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12444.229324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12449.509662] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12453.348324] LustreError: 414033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12453.387745] LustreError: 414033:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12458.471287] LustreError: 414034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12458.773202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12459.377652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12464.616412] LustreError: 414029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12464.645194] LustreError: 414029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12466.116577] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12468.710257] LustreError: 414030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12468.726416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12468.742383] LustreError: 414030:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12473.453369] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12473.832722] LustreError: 414033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12473.850965] LustreError: 414033:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12480.245028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12480.762125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12485.302879] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12494.306344] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12494.309402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12494.318775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12495.439886] Lustre: server umount lustre-OST0000 complete [12501.475529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12501.484085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12501.484763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12504.998757] Lustre: server umount lustre-MDT0000 complete [12506.598233] LustreError: 414033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12506.625856] LustreError: 414033:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12508.806637] LustreError: 414012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385364 with bad export cookie 2602844891681873917 [12508.813967] LustreError: 414012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12508.814728] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12518.557667] Lustre: server umount lustre-MDT0001 complete [12528.974028] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [12531.429887] Key type lgssc unregistered [12531.672568] LNet: 417184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12531.694124] LNetError: 417184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12531.725443] LNet: Removed LNI 192.168.202.118@tcp [12532.538194] Key type .llcrypt unregistered [12532.540670] Key type ._llcrypt unregistered [12558.464945] Key type ._llcrypt registered [12558.467534] Key type .llcrypt registered [12558.572762] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [12559.774285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12559.803033] alg: No test for adler32 (adler32-zlib) [12560.840308] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [12561.176898] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [12562.881641] Key type lgssc registered [12564.133117] Lustre: Echo OBD driver; http://www.lustre.org/ [12572.135581] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12572.145957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12587.680488] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12592.800605] LustreError: 418574:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.118@tcp: failed processing log, type 4: rc = -110 [12619.488233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12625.850675] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12633.968220] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12641.384515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12641.902427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12641.985462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12645.962236] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12647.394245] LustreError: 418600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12647.420049] LustreError: 418600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12652.516725] LustreError: 418601:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12653.545457] LustreError: 419613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12653.557205] LustreError: 419613:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12654.119771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12654.442416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12658.096406] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12665.854842] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12671.788216] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12680.156532] Lustre: server umount lustre-OST0000 complete [12685.282535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12685.311983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12685.316973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12690.402938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12690.414786] Lustre: Skipped 2 previous similar messages [12694.579803] Lustre: server umount lustre-MDT0000 complete [12695.521564] Lustre: 417913:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385535/real 1787385535] req@0000000047bbbb93 x1874209449997312/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787385551 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12695.529876] LustreError: 419617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12695.551186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12695.559890] LustreError: 419617:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12695.582172] Lustre: Skipped 1 previous similar message [12698.291932] LustreError: 418582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385554 with bad export cookie 728450467174620204 [12698.303686] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12698.633120] Lustre: server umount lustre-MDT0001 complete [12710.029603] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [12713.389085] Key type lgssc unregistered [12713.687703] LNet: 422146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12713.697307] LNetError: 422146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12713.723642] LNet: Removed LNI 192.168.202.118@tcp [12715.139111] Key type .llcrypt unregistered [12715.141209] Key type ._llcrypt unregistered [12746.773843] Key type ._llcrypt registered [12746.776710] Key type .llcrypt registered [12746.919657] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [12748.282745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12748.380742] alg: No test for adler32 (adler32-zlib) [12749.469545] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [12749.695426] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [12751.378183] Key type lgssc registered [12752.372739] Lustre: Echo OBD driver; http://www.lustre.org/ [12759.192748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12759.203485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12760.613663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12764.527575] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12770.801733] LustreError: 423568:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12770.829045] LustreError: 423568:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12773.379406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12773.748878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12778.300938] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12785.980489] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12792.418710] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12798.869880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12799.211278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12805.372334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12806.362663] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12813.796625] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12819.301671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12819.309773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12819.335951] Lustre: Skipped 2 previous similar messages [12819.358460] Lustre: Skipped 2 previous similar messages [12821.851628] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.18@tcp (stopping) [12824.549715] LustreError: 423568:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12824.562323] LustreError: 423568:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12824.710457] Lustre: server umount lustre-MDT0000 complete [12827.361407] LustreError: 425411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.18@tcp arrived at 1787385683 with bad export cookie 3119522422533933725 [12827.372969] LustreError: 425411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12829.269934] LustreError: 423550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385685 with bad export cookie 3119522422533932836 [12829.279204] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12829.283626] LustreError: 423550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12829.672703] LustreError: 423569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12829.676956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12829.689437] LustreError: 423569:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12829.706290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12834.786329] LustreError: 423563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12834.787236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12834.798742] LustreError: 423563:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12835.615383] Lustre: server umount lustre-OST0000 complete [12839.906515] LustreError: 423569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12839.917030] LustreError: 423569:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12843.936590] LustreError: 424297:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000972ab8a0 x1874209648181632/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12844.168611] Lustre: server umount lustre-MDT0001 complete [12857.973728] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [12861.405678] Key type lgssc unregistered [12861.837680] LNet: 427267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12861.855336] LNetError: 427267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12861.876912] LNet: Removed LNI 192.168.202.118@tcp [12863.170936] Key type .llcrypt unregistered [12863.173719] Key type ._llcrypt unregistered [12895.683591] Key type ._llcrypt registered [12895.698034] Key type .llcrypt registered [12895.869348] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [12897.234484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12897.309985] alg: No test for adler32 (adler32-zlib) [12898.572627] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [12899.082313] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [12901.104235] Key type lgssc registered [12904.260495] Lustre: Echo OBD driver; http://www.lustre.org/ [12914.475787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12914.502683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12916.488552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12921.865588] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12926.955984] LustreError: 428690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12926.977379] LustreError: 428690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12932.066160] LustreError: 428691:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12932.873952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12933.151286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12937.701816] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12947.407032] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12954.327689] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12961.349308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12961.703902] LustreError: 430514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12962.023985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12967.420377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12970.992605] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12981.138829] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12987.882358] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12987.917974] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12987.924323] Lustre: Skipped 1 previous similar message [12989.203922] Lustre: server umount lustre-MDT0001 complete [12993.022419] LustreError: 428691:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12993.090718] LustreError: 428691:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12998.120548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12998.124588] LustreError: 429371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12998.140491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12998.140505] Lustre: Skipped 1 previous similar message [12998.148739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12998.208628] Lustre: Skipped 1 previous similar message [13003.239886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13004.207646] Lustre: server umount lustre-OST0000 complete [13008.355305] LustreError: 429371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13008.383190] LustreError: 429371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13009.559608] Lustre: server umount lustre-MDT0000 complete [13026.175483] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13029.125184] Key type lgssc unregistered [13029.506924] LNet: 432378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13029.514235] LNetError: 432378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13029.530741] LNet: Removed LNI 192.168.202.118@tcp [13030.211131] Key type .llcrypt unregistered [13030.214656] Key type ._llcrypt unregistered [13057.654657] Key type ._llcrypt registered [13057.656199] Key type .llcrypt registered [13057.745631] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [13076.414185] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [13077.675500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13077.716523] alg: No test for adler32 (adler32-zlib) [13078.778734] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [13078.967849] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [13080.640530] Key type lgssc registered [13081.654770] Lustre: Echo OBD driver; http://www.lustre.org/ [13135.973132] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [13154.772574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13154.803305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13156.133451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13156.220650] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13156.348151] Lustre: lustre-MDT0000: new disk, initializing [13156.521211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13156.543705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13162.074637] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13180.224859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13180.518672] Lustre: lustre-OST0000: new disk, initializing [13180.524351] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13180.529261] Lustre: Skipped 1 previous similar message [13180.540334] Lustre: 437439:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13180.615752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13188.059919] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13188.193187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13188.199873] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13188.232142] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13198.800732] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13216.662983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13216.858698] Lustre: 436420:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13216.966770] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13217.097192] Lustre: lustre-MDT0001: new disk, initializing [13217.246733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13217.290946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13217.301673] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13222.715187] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13232.837843] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13233.068224] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13237.910536] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13238.126318] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13245.923142] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13245.936123] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13245.955373] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13245.960680] Lustre: Skipped 1 previous similar message [13245.980863] Lustre: Skipped 1 previous similar message [13248.369885] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.18@tcp (stopping) [13250.136803] Lustre: server umount lustre-MDT0001 complete [13256.165868] LustreError: 436432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13256.179747] LustreError: 436432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13261.281647] LustreError: 436426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13261.308743] LustreError: 436426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13266.411676] LustreError: 438115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13266.437920] LustreError: 438115:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13271.524968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13271.525404] LustreError: 436432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13271.552329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13276.531437] Lustre: server umount lustre-OST0000 complete [13276.646074] LustreError: 436431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13288.316077] Lustre: server umount lustre-MDT0000 complete [13304.918845] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13308.277211] Key type lgssc unregistered [13308.530602] LNet: 441198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13308.535595] LNetError: 441198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13308.554759] LNet: Removed LNI 192.168.202.118@tcp [13309.779060] Key type .llcrypt unregistered [13309.780826] Key type ._llcrypt unregistered [13366.691855] Key type ._llcrypt registered [13366.697101] Key type .llcrypt registered [13366.867820] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13367.341375] Key type .llcrypt unregistered [13367.343100] Key type ._llcrypt unregistered [13387.770679] Key type ._llcrypt registered [13387.771745] Key type .llcrypt registered [13387.907471] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [13389.364163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13389.399031] alg: No test for adler32 (adler32-zlib) [13390.474562] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [13390.825326] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [13392.568228] Key type lgssc registered [13393.799844] Lustre: Echo OBD driver; http://www.lustre.org/ [13425.961928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13426.032181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13427.483682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13427.542039] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13427.696269] Lustre: lustre-MDT0000: new disk, initializing [13427.901573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13427.916525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13435.459926] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13448.704533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13448.970282] Lustre: 444587:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13449.044361] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13449.055505] Lustre: Skipped 1 previous similar message [13449.150933] Lustre: lustre-MDT0001: new disk, initializing [13449.311655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13449.399292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13449.428236] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13453.304895] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13463.292402] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13470.249820] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13477.761458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13478.016495] Lustre: lustre-OST0000: new disk, initializing [13478.019368] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13478.024911] Lustre: 446556:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13478.109283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13484.116936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13484.137691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13484.235845] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13484.886698] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13495.523462] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13504.997168] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13505.005759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13505.013218] Lustre: Skipped 2 previous similar messages [13505.050706] Lustre: Skipped 2 previous similar messages [13509.058731] Lustre: server umount lustre-MDT0000 complete [13513.386162] LustreError: 444580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386369 with bad export cookie 5583185141648412482 [13513.388518] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13513.393046] LustreError: 444580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13514.035954] Lustre: server umount lustre-MDT0001 complete [13528.792531] Lustre: server umount lustre-OST0000 complete [13540.360414] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 04:13:13 (1787386393) [13548.913466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13549.758188] LustreError: 448585:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13549.797524] LustreError: 448585:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13549.910524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13555.171387] LustreError: 448586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13555.257147] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13559.270045] LustreError: 448585:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13564.388397] LustreError: 448586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13565.132671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13565.641680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13570.167568] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13578.923864] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13585.360500] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13605.292941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13605.528076] Lustre: Found index 0 for lustre-OST0000, updating log [13605.776853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13612.866068] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13621.183332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13621.220418] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13626.338885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13626.342072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13626.345324] LustreError: Skipped 1 previous similar message [13626.350453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13626.352598] Lustre: Skipped 1 previous similar message [13626.367755] Lustre: Skipped 1 previous similar message [13627.430473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13627.436574] Lustre: Skipped 1 previous similar message [13628.614708] Lustre: server umount lustre-OST0000 complete [13647.845162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13647.851161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13647.859411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13651.218839] Lustre: server umount lustre-MDT0000 complete [13653.990286] LustreError: 449798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13654.006212] LustreError: 449798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13654.902913] LustreError: 449321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386510 with bad export cookie 5583185141648413875 [13654.921582] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13655.147518] Lustre: server umount lustre-MDT0001 complete [13663.823873] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 04:15:17 (1787386517) [13666.336286] LNet: 1 local NIs in recovery (showing 1): 192.168.202.118@tcp [14286.173786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14286.710896] LustreError: 454997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14286.808398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14291.940263] LustreError: 454998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14292.054497] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14297.076228] LustreError: 454997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14301.333526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14301.931367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14307.485197] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14316.406603] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14323.455682] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14343.911150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14344.226233] LNet: 1 local NIs in recovery (showing 1): 192.168.202.118@tcp [14344.318830] Lustre: Found index 0 for lustre-OST0000, updating log [14344.843399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14353.964230] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14363.267862] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14365.669067] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14365.683795] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14365.706644] Lustre: Skipped 1 previous similar message [14365.716692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14365.729045] Lustre: Skipped 2 previous similar messages [14371.868795] Lustre: server umount lustre-OST0000 complete [14377.571408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14377.582225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14378.657355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14378.659706] Lustre: Skipped 1 previous similar message [14380.708984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14380.713914] Lustre: Skipped 4 previous similar messages [14382.800445] LustreError: 454993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14382.813372] LustreError: 454993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14382.858588] Lustre: server umount lustre-MDT0000 complete [14386.578771] LustreError: 454979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387242 with bad export cookie 5583185141648414960 [14386.583962] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14386.587311] LustreError: 454979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [14387.035707] Lustre: server umount lustre-MDT0001 complete [14403.618783] LNet: 1 local NIs in recovery (showing 1): 192.168.202.118@tcp [14420.465983] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [14431.298137] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [14469.259181] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 04:28:43 (1787387323) [14492.201077] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [14495.719414] Key type lgssc unregistered [14496.145593] LNet: 461612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14496.155523] LNetError: 461612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14607.832794] LNet: Removed LNI 192.168.202.118@tcp [14609.226330] Key type .llcrypt unregistered [14609.228471] Key type ._llcrypt unregistered [14630.995204] Key type ._llcrypt registered [14630.996533] Key type .llcrypt registered [14631.177613] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [14632.289858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14632.406161] alg: No test for adler32 (adler32-zlib) [14633.617496] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [14633.875287] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [14635.578246] Key type lgssc registered [14637.188927] Lustre: Echo OBD driver; http://www.lustre.org/ [14642.000251] LNet: Added LNI 192.168.202.118@tcp100 [8/256/0/180] [14642.992052] LNet: Added LNI 192.168.202.118@tcp101 [8/256/0/180] [14643.850647] LNet: Added LNI 192.168.202.118@tcp102 [8/256/0/180] [14645.080692] LNet: Added LNI 192.168.202.118@tcp103 [8/256/0/180] [14646.158859] LNet: Added LNI 192.168.202.118@tcp104 [8/256/0/180] [14647.400585] LNet: Added LNI 192.168.202.118@tcp105 [8/256/0/180] [14648.627507] LNet: Added LNI 192.168.202.118@tcp106 [8/256/0/180] [14649.744751] LNet: Added LNI 192.168.202.118@tcp107 [8/256/0/180] [14650.769573] LNet: Added LNI 192.168.202.118@tcp108 [8/256/0/180] [14651.975463] LNet: Added LNI 192.168.202.118@tcp109 [8/256/0/180] [14652.983623] LNet: Added LNI 192.168.202.118@tcp110 [8/256/0/180] [14654.139163] LNet: Added LNI 192.168.202.118@tcp111 [8/256/0/180] [14655.445372] LNet: Added LNI 192.168.202.118@tcp112 [8/256/0/180] [14656.602740] LNet: Added LNI 192.168.202.118@tcp113 [8/256/0/180] [14658.009546] LNet: Added LNI 192.168.202.118@tcp114 [8/256/0/180] [14659.205435] LNet: Added LNI 192.168.202.118@tcp115 [8/256/0/180] [14660.309284] LNet: Added LNI 192.168.202.118@tcp116 [8/256/0/180] [14661.333546] LNet: Added LNI 192.168.202.118@tcp117 [8/256/0/180] [14662.461303] LNet: Added LNI 192.168.202.118@tcp118 [8/256/0/180] [14663.351377] LNet: Added LNI 192.168.202.118@tcp119 [8/256/0/180] [14664.641220] LNet: Added LNI 192.168.202.118@tcp120 [8/256/0/180] [14665.813293] LNet: Added LNI 192.168.202.118@tcp121 [8/256/0/180] [14667.209552] LNet: Added LNI 192.168.202.118@tcp122 [8/256/0/180] [14668.336647] LNet: Added LNI 192.168.202.118@tcp123 [8/256/0/180] [14669.506763] LNet: Added LNI 192.168.202.118@tcp124 [8/256/0/180] [14670.920109] LNet: Added LNI 192.168.202.118@tcp125 [8/256/0/180] [14671.717314] LNet: Added LNI 192.168.202.118@tcp126 [8/256/0/180] [14672.854804] LNet: Added LNI 192.168.202.118@tcp127 [8/256/0/180] [14674.123316] LNet: Added LNI 192.168.202.118@tcp128 [8/256/0/180] [14675.048525] LNet: Added LNI 192.168.202.118@tcp129 [8/256/0/180] [14676.444266] LNet: Added LNI 192.168.202.118@tcp130 [8/256/0/180] [14677.625379] LNet: Added LNI 192.168.202.118@tcp131 [8/256/0/180] [14678.735975] LNet: Added LNI 192.168.202.118@tcp132 [8/256/0/180] [14679.864579] LNet: Added LNI 192.168.202.118@tcp133 [8/256/0/180] [14681.046429] LNet: Added LNI 192.168.202.118@tcp134 [8/256/0/180] [14682.142928] LNet: Added LNI 192.168.202.118@tcp135 [8/256/0/180] [14689.446513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14689.490708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14690.667769] Lustre: 464650:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.118@tcp: can't add NID 192.168.202.118@tcp130: rc = -75 [14690.770196] LNetError: 462107:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [14690.784754] LNetError: Protocol error connecting to 192.168.202.118@tcp116 at host 192.168.202.118:988: is it running a compatible version of Lustre? [14690.917153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14690.937042] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14690.995660] Lustre: lustre-MDT0000: new disk, initializing [14691.057897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14691.072648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14691.360204] LNet: 2 local NIs in recovery (showing 2): 192.168.202.118@tcp128, 192.168.202.118@tcp129 [14691.361857] LNetError: 464673:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [14691.393119] LNetError: 464673:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [14691.406155] LNetError: Protocol error connecting to 192.168.202.118@tcp101 at host 192.168.202.118:988: is it running a compatible version of Lustre? [14691.426650] LNetError: Skipped 6 previous similar messages [14696.614426] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14710.001884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14710.176862] Lustre: 464665:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14710.242643] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14710.245453] Lustre: Skipped 1 previous similar message [14710.315702] Lustre: lustre-MDT0001: new disk, initializing [14710.375770] Lustre: 465468:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.118@tcp: can't add NID 192.168.202.118@tcp130: rc = -75 [14710.383755] Lustre: 465468:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14710.528551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14710.576653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14710.591756] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14715.778391] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14718.002886] Lustre: 465832:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.118@tcp: can't add NID 192.168.202.118@tcp130: rc = -75 [14718.019665] Lustre: 465832:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14720.932100] LNetError: 464676:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [14720.938072] LNetError: Protocol error connecting to 192.168.202.118@tcp124 at host 192.168.202.118:988: is it running a compatible version of Lustre? [14722.977866] LNetError: 464683:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [14722.998733] LNetError: 464683:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14723.010537] LNetError: Protocol error connecting to 192.168.202.118@tcp101 at host 192.168.202.118:988: is it running a compatible version of Lustre? [14723.034227] LNetError: Skipped 1 previous similar message [14726.227597] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14729.127426] LNetError: 462106:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [14729.145349] LNetError: Protocol error connecting to 192.168.202.118@tcp118 at host 192.168.202.118:988: is it running a compatible version of Lustre? [14733.606713] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14741.412510] LNetError: 464670:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [14741.427180] LNetError: 464670:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [14741.439890] LNetError: Protocol error connecting to 192.168.202.118@tcp118 at host 192.168.202.118:988: is it running a compatible version of Lustre? [14741.459695] LNetError: Skipped 5 previous similar messages [14742.361916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14742.785367] Lustre: lustre-OST0000: new disk, initializing [14742.787914] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14742.801449] Lustre: 466642:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14742.922157] Lustre: 466647:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.118@tcp: can't add NID 192.168.202.118@tcp130: rc = -75 [14742.928675] Lustre: 466647:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14742.960710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14749.125253] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14752.276973] Lustre: 467116:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.118@tcp: can't add NID 192.168.202.118@tcp130: rc = -75 [14752.291384] Lustre: 467116:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14752.366115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14752.371357] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14752.480345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14757.799436] LNetError: 462107:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [14757.808527] LNetError: 462107:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14757.819336] LNetError: Protocol error connecting to 192.168.202.118@tcp116 at host 192.168.202.118:988: is it running a compatible version of Lustre? [14757.839983] LNetError: Skipped 7 previous similar messages [14758.408952] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14768.097453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14768.104688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14768.113201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14768.119958] Lustre: Skipped 1 previous similar message [14768.128962] Lustre: Skipped 1 previous similar message [14771.800169] Lustre: server umount lustre-OST0000 complete [14777.314233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14777.332788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14777.358639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14778.339101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14778.373761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14778.386953] Lustre: Skipped 1 previous similar message [14782.423300] Lustre: server umount lustre-MDT0000 complete [14786.258663] LustreError: 464660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387642 with bad export cookie 16045987427742096408 [14786.279860] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14786.735645] Lustre: server umount lustre-MDT0001 complete [14793.635366] LNetError: 462109:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [14793.647462] LNetError: 462109:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [14793.652706] LNetError: Protocol error connecting to 192.168.202.118@tcp118 at host 192.168.202.118:988: is it running a compatible version of Lustre? [14793.658840] LNetError: Skipped 8 previous similar messages [14798.658714] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [14801.874231] Key type lgssc unregistered [14802.358801] LNet: 468559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14802.364727] LNetError: 468559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14803.425519] LNet: Removed LNI 192.168.202.118@tcp [14803.430546] LNetError: 468559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14803.441042] LNet: Removed LNI 192.168.202.118@tcp100 [14804.448657] LNet: Removed LNI 192.168.202.118@tcp101 [14804.453244] LNetError: 468559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14804.461267] LNetError: 468559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14804.478499] LNet: Removed LNI 192.168.202.118@tcp102 [14805.536206] LNet: Removed LNI 192.168.202.118@tcp103 [14806.624197] LNet: Removed LNI 192.168.202.118@tcp104 [14806.626601] LNetError: 468559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14806.642629] LNetError: 468559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14806.668660] LNet: Removed LNI 192.168.202.118@tcp105 [14807.712230] LNet: Removed LNI 192.168.202.118@tcp106 [14808.736318] LNet: Removed LNI 192.168.202.118@tcp107 [14809.761218] LNet: Removed LNI 192.168.202.118@tcp108 [14810.784296] LNet: Removed LNI 192.168.202.118@tcp109 [14810.791668] LNetError: 468559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14810.802980] LNetError: 468559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14810.826152] LNet: Removed LNI 192.168.202.118@tcp110 [14811.873124] LNet: Removed LNI 192.168.202.118@tcp111 [14812.896797] LNet: Removed LNI 192.168.202.118@tcp112 [14813.920804] LNet: Removed LNI 192.168.202.118@tcp113 [14814.944288] LNet: Removed LNI 192.168.202.118@tcp114 [14815.969530] LNet: Removed LNI 192.168.202.118@tcp115 [14816.992164] LNet: Removed LNI 192.168.202.118@tcp116 [14818.016315] LNet: Removed LNI 192.168.202.118@tcp117 [14819.042150] LNet: Removed LNI 192.168.202.118@tcp118 [14819.043955] LNetError: 468559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14819.057391] LNetError: 468559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14819.091658] LNet: Removed LNI 192.168.202.118@tcp119 [14820.132436] LNet: Removed LNI 192.168.202.118@tcp120 [14821.152204] LNet: Removed LNI 192.168.202.118@tcp121 [14822.176192] LNet: Removed LNI 192.168.202.118@tcp122 [14823.258493] LNet: Removed LNI 192.168.202.118@tcp123 [14824.292666] LNet: Removed LNI 192.168.202.118@tcp124 [14825.312262] LNet: Removed LNI 192.168.202.118@tcp125 [14826.336156] LNet: Removed LNI 192.168.202.118@tcp126 [14827.360328] LNet: Removed LNI 192.168.202.118@tcp127 [14828.384281] LNet: Removed LNI 192.168.202.118@tcp128 [14829.410296] LNet: Removed LNI 192.168.202.118@tcp129 [14829.413578] LNet: Removed LNI 192.168.202.118@tcp130 [14829.419481] LNet: Removed LNI 192.168.202.118@tcp131 [14829.427032] LNet: Removed LNI 192.168.202.118@tcp132 [14829.433772] LNet: Removed LNI 192.168.202.118@tcp133 [14829.440327] LNet: Removed LNI 192.168.202.118@tcp134 [14829.462448] LNet: Removed LNI 192.168.202.118@tcp135 [14830.542958] Key type .llcrypt unregistered [14830.545300] Key type ._llcrypt unregistered [14861.699602] Key type ._llcrypt registered [14861.702657] Key type .llcrypt registered [14861.886628] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [14879.051934] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [14881.232815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14881.305408] alg: No test for adler32 (adler32-zlib) [14882.469671] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [14882.797061] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [14884.472202] Key type lgssc registered [14885.742549] Lustre: Echo OBD driver; http://www.lustre.org/ [14923.903270] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 04:36:17 (1787387777) [14944.362109] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [14947.167201] Key type lgssc unregistered [14947.508212] LNet: 472652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14947.516518] LNetError: 472652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14947.537415] LNet: Removed LNI 192.168.202.118@tcp [14948.476362] Key type .llcrypt unregistered [14948.478796] Key type ._llcrypt unregistered [14966.068036] Key type ._llcrypt registered [14966.070198] Key type .llcrypt registered [14966.159936] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [14967.336503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14967.349965] alg: No test for adler32 (adler32-zlib) [14968.444300] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [14968.744815] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [14970.416168] Key type lgssc registered [14971.462585] Lustre: Echo OBD driver; http://www.lustre.org/ [14980.254742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14980.291103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14981.526350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14981.548954] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14981.632321] Lustre: lustre-MDT0000: new disk, initializing [14981.705325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14981.719241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14986.014928] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14996.867650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14997.025801] Lustre: 473867:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14997.081667] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14997.086552] Lustre: Skipped 1 previous similar message [14997.164826] Lustre: lustre-MDT0001: new disk, initializing [14997.277819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14997.334016] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14997.353492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15002.379774] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15011.890738] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15017.783478] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15024.612693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15025.027337] Lustre: lustre-OST0000: new disk, initializing [15025.031549] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15025.038935] Lustre: 475837:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15025.120372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15031.033153] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15034.445251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15034.458583] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15034.615606] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15040.807109] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15045.166232] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing lnet_if_list [15050.858939] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15050.901170] LNet: Added LNI 192.168.202.118@tcp10 [8/256/0/180] [15058.195318] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15058.226334] LNet: Added LNI 192.168.202.118@tcp11 [8/256/0/180] [15063.638248] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15063.652342] LNet: Added LNI 192.168.202.118@tcp12 [8/256/0/180] [15069.652638] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15069.675689] LNet: Added LNI 192.168.202.118@tcp13 [8/256/0/180] [15075.081731] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15075.115035] LNet: Added LNI 192.168.202.118@tcp14 [8/256/0/180] [15080.021940] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15080.085862] LNet: Added LNI 192.168.202.118@tcp15 [8/256/0/180] [15086.731272] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15086.790997] LNet: Added LNI 192.168.202.118@tcp16 [8/256/0/180] [15092.435584] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15092.538426] LNet: Added LNI 192.168.202.118@tcp17 [8/256/0/180] [15098.220609] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15098.276632] LNet: Added LNI 192.168.202.118@tcp18 [8/256/0/180] [15103.069461] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15103.092232] LNet: Added LNI 192.168.202.118@tcp19 [8/256/0/180] [15108.317387] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15108.339500] LNet: Added LNI 192.168.202.118@tcp20 [8/256/0/180] [15114.378507] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15114.418749] LNet: Added LNI 192.168.202.118@tcp21 [8/256/0/180] [15120.736770] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15120.766514] LNet: Added LNI 192.168.202.118@tcp22 [8/256/0/180] [15127.553703] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15127.592458] LNet: Added LNI 192.168.202.118@tcp23 [8/256/0/180] [15133.316056] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15133.340828] LNet: Added LNI 192.168.202.118@tcp24 [8/256/0/180] [15140.250345] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15140.276216] LNet: Added LNI 192.168.202.118@tcp25 [8/256/0/180] [15146.696837] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15146.731661] LNet: Added LNI 192.168.202.118@tcp26 [8/256/0/180] [15153.235270] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15153.279505] LNet: Added LNI 192.168.202.118@tcp27 [8/256/0/180] [15159.235703] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15159.274084] LNet: Added LNI 192.168.202.118@tcp28 [8/256/0/180] [15166.418500] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15166.469261] LNet: Added LNI 192.168.202.118@tcp29 [8/256/0/180] [15173.027967] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15173.079047] LNet: Added LNI 192.168.202.118@tcp30 [8/256/0/180] [15180.715435] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15180.756445] LNet: Added LNI 192.168.202.118@tcp31 [8/256/0/180] [15188.153869] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15188.167954] LNet: Added LNI 192.168.202.118@tcp32 [8/256/0/180] [15197.127669] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15197.170443] LNet: Added LNI 192.168.202.118@tcp33 [8/256/0/180] [15205.562077] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15205.594228] LNet: Added LNI 192.168.202.118@tcp34 [8/256/0/180] [15213.913954] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15213.957169] LNet: Added LNI 192.168.202.118@tcp35 [8/256/0/180] [15222.016375] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15222.057548] LNet: Added LNI 192.168.202.118@tcp36 [8/256/0/180] [15230.631789] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15230.695087] LNet: Added LNI 192.168.202.118@tcp37 [8/256/0/180] [15240.102943] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15240.128481] LNet: Added LNI 192.168.202.118@tcp38 [8/256/0/180] [15250.904120] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15250.965820] LNet: Added LNI 192.168.202.118@tcp39 [8/256/0/180] [15260.585979] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15260.624892] LNet: Added LNI 192.168.202.118@tcp40 [8/256/0/180] [15269.944715] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15270.026072] LNet: Added LNI 192.168.202.118@tcp41 [8/256/0/180] [15279.500933] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15279.537416] LNet: Added LNI 192.168.202.118@tcp42 [8/256/0/180] [15289.851900] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15289.873830] LNet: Added LNI 192.168.202.118@tcp43 [8/256/0/180] [15300.497630] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15300.578127] LNet: Added LNI 192.168.202.118@tcp44 [8/256/0/180] [15309.513474] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15309.551478] LNet: Added LNI 192.168.202.118@tcp45 [8/256/0/180] [15318.780884] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15318.806593] LNet: Added LNI 192.168.202.118@tcp46 [8/256/0/180] [15327.404549] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15327.437832] LNet: Added LNI 192.168.202.118@tcp47 [8/256/0/180] [15336.349341] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15336.375983] LNet: Added LNI 192.168.202.118@tcp48 [8/256/0/180] [15345.801786] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15345.861550] LNet: Added LNI 192.168.202.118@tcp49 [8/256/0/180] [15355.052262] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15355.076793] LNet: Added LNI 192.168.202.118@tcp50 [8/256/0/180] [15376.369480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15376.384305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15376.416214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15377.890642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15377.892722] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15377.912202] Lustre: Skipped 1 previous similar message [15381.520725] Lustre: server umount lustre-MDT0000 complete [15386.373119] LustreError: 473861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388242 with bad export cookie 4166446093744439069 [15386.383308] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15386.905522] Lustre: server umount lustre-MDT0001 complete [15404.256765] Lustre: 473141:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787388243/real 1787388243] req@00000000b56a9903 x1874211975122688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787388259 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15404.294893] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15406.561120] Lustre: 473143:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787388246/real 1787388246] req@00000000b2bf7546 x1874211975122816/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787388262 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15408.618030] Lustre: 473142:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787388248/real 1787388248] req@00000000ae784e8c x1874211975122944/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787388264 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15411.917391] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [15412.705713] Lustre: 473140:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787388252/real 1787388252] req@00000000ec85aa22 x1874211975123200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787388268 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15417.890087] Lustre: 473140:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787388257/real 1787388257] req@00000000e6ca686a x1874211975123328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787388273 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15421.147478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15432.319541] LustreError: 475829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15432.367935] LustreError: 475829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15432.405177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15433.463950] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15433.464708] LustreError: 475830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15434.466932] LustreError: 475829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15437.652995] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15439.847873] LustreError: 475830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15444.969205] LustreError: 475829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15447.210371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15447.630835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15449.724155] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15452.317251] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15461.249766] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15467.672654] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15491.046163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15491.059791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15491.072346] Lustre: Skipped 3 previous similar messages [15496.184593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15496.191692] Lustre: Skipped 1 previous similar message [15501.283126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15505.376447] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15505.691655] Lustre: server umount lustre-OST0000 complete [15511.523927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15511.526532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15511.545899] Lustre: Skipped 2 previous similar messages [15511.573514] Lustre: Skipped 2 previous similar messages [15524.320158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15524.665840] Lustre: server umount lustre-MDT0000 complete [15526.887392] LustreError: 500087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15526.924773] LustreError: 500087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15529.464117] LustreError: 498944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388385 with bad export cookie 4166446093744441736 [15529.489823] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15530.332380] Lustre: server umount lustre-MDT0001 complete [15543.032437] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [15546.328223] Key type lgssc unregistered [15546.875546] LNet: 502890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15546.883775] LNetError: 502890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15547.937229] LNet: Removed LNI 192.168.202.118@tcp [15547.939123] LNetError: 502890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15547.957047] LNet: Removed LNI 192.168.202.118@tcp10 [15547.968764] LNet: Removed LNI 192.168.202.118@tcp11 [15547.976460] LNet: Removed LNI 192.168.202.118@tcp12 [15547.991662] LNet: Removed LNI 192.168.202.118@tcp13 [15547.998145] LNet: Removed LNI 192.168.202.118@tcp14 [15548.016300] LNet: Removed LNI 192.168.202.118@tcp15 [15548.024787] LNet: Removed LNI 192.168.202.118@tcp16 [15548.030155] LNet: Removed LNI 192.168.202.118@tcp17 [15548.047928] LNet: Removed LNI 192.168.202.118@tcp18 [15548.053290] LNet: Removed LNI 192.168.202.118@tcp19 [15548.064308] LNet: Removed LNI 192.168.202.118@tcp20 [15548.079133] LNet: Removed LNI 192.168.202.118@tcp21 [15548.085992] LNet: Removed LNI 192.168.202.118@tcp22 [15548.094924] LNet: Removed LNI 192.168.202.118@tcp23 [15548.100463] LNet: Removed LNI 192.168.202.118@tcp24 [15548.108518] LNet: Removed LNI 192.168.202.118@tcp25 [15548.114773] LNet: Removed LNI 192.168.202.118@tcp26 [15548.121647] LNet: Removed LNI 192.168.202.118@tcp27 [15548.127491] LNet: Removed LNI 192.168.202.118@tcp28 [15548.136489] LNet: Removed LNI 192.168.202.118@tcp29 [15548.142878] LNet: Removed LNI 192.168.202.118@tcp30 [15548.148619] LNet: Removed LNI 192.168.202.118@tcp31 [15548.156735] LNet: Removed LNI 192.168.202.118@tcp32 [15548.164517] LNet: Removed LNI 192.168.202.118@tcp33 [15548.168578] LNet: Removed LNI 192.168.202.118@tcp34 [15548.178258] LNet: Removed LNI 192.168.202.118@tcp35 [15548.185878] LNet: Removed LNI 192.168.202.118@tcp36 [15548.195735] LNet: Removed LNI 192.168.202.118@tcp37 [15548.207434] LNet: Removed LNI 192.168.202.118@tcp38 [15548.214162] LNet: Removed LNI 192.168.202.118@tcp39 [15548.222494] LNet: Removed LNI 192.168.202.118@tcp40 [15548.226921] LNet: Removed LNI 192.168.202.118@tcp41 [15548.230215] LNet: Removed LNI 192.168.202.118@tcp42 [15548.236950] LNet: Removed LNI 192.168.202.118@tcp43 [15548.238765] LNet: Removed LNI 192.168.202.118@tcp44 [15548.248172] LNet: Removed LNI 192.168.202.118@tcp45 [15548.249795] LNet: Removed LNI 192.168.202.118@tcp46 [15548.257352] LNet: Removed LNI 192.168.202.118@tcp47 [15548.259268] LNet: Removed LNI 192.168.202.118@tcp48 [15548.273772] LNet: Removed LNI 192.168.202.118@tcp49 [15548.298347] LNet: Removed LNI 192.168.202.118@tcp50 [15549.271254] Key type .llcrypt unregistered [15549.275701] Key type ._llcrypt unregistered [15574.898128] Key type ._llcrypt registered [15574.901884] Key type .llcrypt registered [15575.065171] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [15591.170509] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [15592.225296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15592.295092] alg: No test for adler32 (adler32-zlib) [15593.424535] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [15593.688812] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [15595.401093] Key type lgssc registered [15596.827230] Lustre: Echo OBD driver; http://www.lustre.org/ [15640.671451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15640.700256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15642.060693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15642.146268] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15642.259491] Lustre: lustre-MDT0000: new disk, initializing [15642.343393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15642.366428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15647.497596] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15659.987219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15660.174787] Lustre: 506088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15660.245462] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15660.253586] Lustre: Skipped 1 previous similar message [15660.387996] Lustre: lustre-MDT0001: new disk, initializing [15660.556787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15660.621431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15660.637577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15666.124413] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15677.507628] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15685.389366] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15693.608577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15693.725670] Lustre: *** cfs_fail_loc=90f, val=0*** [15693.778736] Lustre: lustre-OST0000: new disk, initializing [15693.781880] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15693.790335] Lustre: 508108:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15693.869310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15699.083095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15699.096149] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15699.295412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15701.826582] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15713.190034] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15725.026030] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15725.040499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15725.043658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15725.056113] Lustre: Skipped 1 previous similar message [15725.081699] Lustre: Skipped 1 previous similar message [15730.162779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15730.179586] Lustre: Skipped 1 previous similar message [15735.274200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15735.286902] Lustre: Skipped 1 previous similar message [15738.336388] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15738.758992] Lustre: server umount lustre-OST0000 complete [15745.508292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15745.523683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15750.240699] Lustre: server umount lustre-MDT0000 complete [15754.519517] LustreError: 508107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388610 with bad export cookie 10391278355673474439 [15754.545772] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15754.549340] LustreError: 508107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15754.909686] Lustre: server umount lustre-MDT0001 complete [15769.669453] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [15773.576396] Key type lgssc unregistered [15773.923535] LNet: 510021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15773.934984] LNetError: 510021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15773.954398] LNet: Removed LNI 192.168.202.118@tcp [15775.194395] Key type .llcrypt unregistered [15775.196569] Key type ._llcrypt unregistered [15808.780556] Key type ._llcrypt registered [15808.782032] Key type .llcrypt registered [15808.975838] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [15810.114356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15810.231769] alg: No test for adler32 (adler32-zlib) [15811.973155] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [15812.315737] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [15814.096987] Key type lgssc registered [15816.026641] Lustre: Echo OBD driver; http://www.lustre.org/ [15827.468602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15827.482039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15828.936123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15833.969548] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15839.205503] LustreError: 511292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15839.237850] LustreError: 511292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15844.332907] LustreError: 511293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15844.743940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15845.312325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15850.974809] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15861.122616] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15868.865811] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15877.835057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15878.240197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15886.507609] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15896.295693] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15915.356622] loop0: detected capacity change from 0 to 204800000 [15915.662171] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15915.685035] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15915.707032] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15923.803970] loop0: detected capacity change from 0 to 204800000 [15923.989909] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15924.010666] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15924.028315] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15929.864662] loop0: detected capacity change from 0 to 204800000 [15937.593565] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15937.785858] Lustre: fs15246-OST0000: new disk, initializing [15937.795038] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15937.809789] Lustre: 514908:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15937.913371] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15945.929897] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15957.031512] loop1: detected capacity change from 0 to 204800000 [15963.178988] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15963.313121] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15963.358809] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15963.471369] Lustre: fs15246-MDT0000: new disk, initializing [15963.691606] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15963.768412] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15963.937161] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15964.988808] Lustre: 516032:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000f3f0055b x1874212859991424/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787388836 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15968.924792] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15970.191048] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15970.201762] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15970.250678] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15995.366985] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15995.378081] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15998.027153] Lustre: server umount fs15246-MDT0000 complete [16000.504677] LustreError: 511288:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16005.628140] LustreError: 511289:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16007.968064] Lustre: server umount fs15246-OST0000 complete [16020.974123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16020.979592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16021.011754] Lustre: Skipped 1 previous similar message [16021.037443] Lustre: Skipped 1 previous similar message [16025.814479] Lustre: server umount lustre-OST0000 complete [16029.666012] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16029.672425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16029.688844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16035.895494] Lustre: server umount lustre-MDT0000 complete [16036.329322] LustreError: 513558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16036.353730] LustreError: 513558:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16040.593034] LustreError: 518382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388896 with bad export cookie 8299308201781974384 [16040.597899] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16040.601450] LustreError: 518382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16040.860668] Lustre: server umount lustre-MDT0001 complete [16057.650851] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [16061.199192] Key type lgssc unregistered [16061.595309] LNet: 519136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16061.620967] LNetError: 519136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16061.652309] LNet: Removed LNI 192.168.202.118@tcp [16063.108390] Key type .llcrypt unregistered [16063.112661] Key type ._llcrypt unregistered [16087.895577] Key type ._llcrypt registered [16087.915480] Key type .llcrypt registered [16088.114785] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [16090.241899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16090.287558] alg: No test for adler32 (adler32-zlib) [16091.699775] Lustre: Lustre: Build Version: 2.17.57_46_g5381a3b [16091.992414] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [16093.728234] Key type lgssc registered [16095.664707] Lustre: Echo OBD driver; http://www.lustre.org/ [16141.380866] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [16154.526484] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [16201.091203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16201.126247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16202.447382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16202.491879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16202.678096] Lustre: lustre-MDT0000: new disk, initializing [16202.842847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16202.876150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16208.386728] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [16221.173778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16221.318895] Lustre: 523606:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16221.369340] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16221.374612] Lustre: Skipped 1 previous similar message [16221.494267] Lustre: lustre-MDT0001: new disk, initializing [16221.581199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16221.609572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16221.613181] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16226.892977] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [16238.795526] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16246.650621] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16255.599086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16256.014713] Lustre: lustre-OST0000: new disk, initializing [16256.032091] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16256.044363] Lustre: 525579:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16256.180676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16261.248441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16261.257637] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16261.441645] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16263.643933] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [16275.165906] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16281.866879] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16282.274526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16288.360593] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16288.666317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16292.322907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16292.336986] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16292.339242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16292.349663] LustreError: Skipped 1 previous similar message [16297.047349] Lustre: server umount lustre-OST0000 complete [16302.561414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16302.564612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16302.587399] Lustre: Skipped 2 previous similar messages [16302.606107] Lustre: Skipped 2 previous similar messages [16307.682685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16307.689911] Lustre: Skipped 1 previous similar message [16308.252038] Lustre: server umount lustre-MDT0000 complete [16312.688496] LustreError: 525578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389168 with bad export cookie 17385386337627495328 [16312.689878] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16312.697045] LustreError: 525578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16313.091773] Lustre: server umount lustre-MDT0001 complete [16345.534354] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [16358.146965] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [16398.618211] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16025 sec ========= 05:00:51 (1787389251) [16400.426266] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:00:54 (1787389254) === [16404.767324] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:00:58 (1787389258) === [16451.744749] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [16455.181500] Key type lgssc unregistered [16455.554529] LNet: 532241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16455.559696] LNetError: 532241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16455.581341] LNet: Removed LNI 192.168.202.118@tcp [16456.555539] Key type .llcrypt unregistered [16456.557701] Key type ._llcrypt unregistered