[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 475213834 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003042] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007807] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010135] LSM: Security Framework initializing [ 0.012043] Yama: becoming mindful. [ 0.013033] SELinux: Initializing. [ 0.014059] *** VALIDATE selinux *** [ 0.022679] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026889] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028100] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030119] *** VALIDATE tmpfs *** [ 0.031473] *** VALIDATE proc *** [ 0.033065] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035261] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036138] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038023] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042196] debug: unmapping init [mem 0xffffffffa2a59000-0xffffffffa2a60fff] [ 0.044163] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045611] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046021] ... version: 2 [ 0.047009] ... bit width: 48 [ 0.047992] ... generic registers: 4 [ 0.048008] ... value mask: 0000ffffffffffff [ 0.049010] ... max period: 00007fffffffffff [ 0.050009] ... fixed-purpose events: 3 [ 0.051000] ... event mask: 000000070000000f [ 0.052307] rcu: Hierarchical SRCU implementation. [ 0.054417] smp: Bringing up secondary CPUs ... [ 0.055593] x86: Booting SMP configuration: [ 0.056031] .... node #0, CPUs: #1 #2 #3 [ 0.060014] smp: Brought up 1 node, 4 CPUs [ 0.061917] smpboot: Max logical packages: 1 [ 0.062014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.149059] node 0 deferred pages initialised in 84ms [ 0.152127] devtmpfs: initialized [ 0.153320] x86/mm: Memory block size: 128MB [ 0.156836] gcov: version magic: 0x41383552 [ 0.159146] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.162160] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.165325] pinctrl core: initialized pinctrl subsystem [ 0.167216] [ 0.167699] ************************************************************* [ 0.170012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173010] ** ** [ 0.175011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.178022] ** ** [ 0.181014] ** This means that this kernel is built to expose internal ** [ 0.183021] ** IOMMU data structures, which may compromise security on ** [ 0.186019] ** your system. ** [ 0.188012] ** ** [ 0.191012] ** If you see this message and you are not debugging the ** [ 0.193032] ** kernel, report this immediately to your vendor! ** [ 0.196014] ** ** [ 0.199011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.201010] ************************************************************* [ 0.204768] NET: Registered protocol family 16 [ 0.206367] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.209063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.212065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.217036] cpuidle: using governor menu [ 0.218713] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.219464] PCI: Using configuration type 1 for base access [ 0.220119] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.228047] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.229020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.231062] cryptd: max_cpu_qlen set to 1000 [ 0.235139] ACPI: Added _OSI(Module Device) [ 0.236015] ACPI: Added _OSI(Processor Device) [ 0.237011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.238010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.241517] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.249598] ACPI: Interpreter enabled [ 0.251061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.252011] ACPI: Using IOAPIC for interrupt routing [ 0.254098] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.257423] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.267810] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.269051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.272020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.276136] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.281414] acpiphp: Slot [2] registered [ 0.283201] acpiphp: Slot [5] registered [ 0.284133] acpiphp: Slot [6] registered [ 0.286123] acpiphp: Slot [7] registered [ 0.287117] acpiphp: Slot [8] registered [ 0.289115] acpiphp: Slot [9] registered [ 0.290176] acpiphp: Slot [10] registered [ 0.292113] acpiphp: Slot [3] registered [ 0.293165] acpiphp: Slot [4] registered [ 0.295128] acpiphp: Slot [11] registered [ 0.296115] acpiphp: Slot [12] registered [ 0.298086] acpiphp: Slot [13] registered [ 0.300130] acpiphp: Slot [14] registered [ 0.301131] acpiphp: Slot [15] registered [ 0.303104] acpiphp: Slot [16] registered [ 0.304104] acpiphp: Slot [17] registered [ 0.306175] acpiphp: Slot [18] registered [ 0.307105] acpiphp: Slot [19] registered [ 0.309147] acpiphp: Slot [20] registered [ 0.311128] acpiphp: Slot [21] registered [ 0.312100] acpiphp: Slot [22] registered [ 0.314167] acpiphp: Slot [23] registered [ 0.316116] acpiphp: Slot [24] registered [ 0.318175] acpiphp: Slot [25] registered [ 0.319094] acpiphp: Slot [26] registered [ 0.321096] acpiphp: Slot [27] registered [ 0.322142] acpiphp: Slot [28] registered [ 0.324101] acpiphp: Slot [29] registered [ 0.326115] acpiphp: Slot [30] registered [ 0.327190] acpiphp: Slot [31] registered [ 0.329068] PCI host bridge to bus 0000:00 [ 0.331061] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.333031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.336028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.339026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.341094] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.344028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.346234] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.348958] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.352049] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.360016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.364041] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.367018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.369021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.371016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.373402] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.376638] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.378039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.380638] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.385014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.397018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.402829] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.407121] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.425018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.437025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.456020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.470806] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.500020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.517021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.551019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.561384] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.582019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.591022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.609016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.620839] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.633018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.640014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.663017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.675724] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.682027] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.692018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.715022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.723997] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.732032] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.740019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.754026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.764809] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.767465] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.770329] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.772512] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.775239] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.780085] iommu: Default domain type: Passthrough [ 0.782881] SCSI subsystem initialized [ 0.784123] ACPI: bus type USB registered [ 0.786087] usbcore: registered new interface driver usbfs [ 0.788090] usbcore: registered new interface driver hub [ 0.790092] usbcore: registered new device driver usb [ 0.791104] pps_core: LinuxPPS API ver. 1 registered [ 0.792007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.794040] PTP clock support registered [ 0.795127] EDAC MC: Ver: 3.0.0 [ 0.796335] PCI: Using ACPI for IRQ routing [ 0.797879] NetLabel: Initializing [ 0.798010] NetLabel: domain hash size = 128 [ 0.799010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.800079] NetLabel: unlabeled traffic allowed by default [ 0.801090] vgaarb: loaded [ 0.803229] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.804009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.811004] clocksource: Switched to clocksource kvm-clock [ 0.905172] VFS: Disk quotas dquot_6.6.0 [ 0.906653] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.909263] *** VALIDATE ramfs *** [ 0.910566] *** VALIDATE hugetlbfs *** [ 0.912256] pnp: PnP ACPI init [ 0.914930] pnp: PnP ACPI: found 6 devices [ 0.944343] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.946797] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.948346] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.950170] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.952042] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.954196] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.956829] NET: Registered protocol family 2 [ 0.958642] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.962030] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.964782] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.970679] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.974295] TCP: Hash tables configured (established 65536 bind 65536) [ 0.977067] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.980027] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.982027] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.984350] NET: Registered protocol family 1 [ 0.987242] RPC: Registered named UNIX socket transport module. [ 0.989241] RPC: Registered udp transport module. [ 0.990623] RPC: Registered tcp transport module. [ 0.992191] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.994311] NET: Registered protocol family 44 [ 0.995856] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.997805] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.999636] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.001850] PCI: CLS 0 bytes, default 64 [ 1.003364] Unpacking initramfs... [ 2.433374] debug: unmapping init [mem 0xffff92fcbcc54000-0xffff92fcbffbffff] [ 2.440159] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.442279] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.445691] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.939904] Initialise system trusted keyrings [ 2.941567] Key type blacklist registered [ 2.944837] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.953695] zbud: loaded [ 2.956137] *** VALIDATE nfs *** [ 2.957276] *** VALIDATE nfs4 *** [ 2.958760] pstore: using deflate compression [ 2.961391] Platform Keyring initialized [ 3.038230] NET: Registered protocol family 38 [ 3.039467] Key type asymmetric registered [ 3.040621] Asymmetric key parser 'x509' registered [ 3.042111] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.044444] io scheduler mq-deadline registered [ 3.045623] io scheduler kyber registered [ 3.046767] io scheduler bfq registered [ 3.047908] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.049894] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.051925] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.053950] ACPI: Power Button [PWRF] [ 3.058980] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.064040] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.076764] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.084246] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.100576] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.129450] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.156414] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.159543] Non-volatile memory driver v1.3 [ 3.160710] Linux agpgart interface v0.103 [ 3.184946] virtio_blk virtio1: [vda] 150616 512-byte logical blocks (77.1 MB/73.5 MiB) [ 3.189194] vda: detected capacity change from 0 to 77115392 [ 3.202449] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.204533] vdb: detected capacity change from 0 to 1073741824 [ 3.217260] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.219873] vdc: detected capacity change from 0 to 2621440000 [ 3.232462] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.234314] vdd: detected capacity change from 0 to 2621440000 [ 3.247898] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.249719] vde: detected capacity change from 0 to 4294967296 [ 3.261067] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.263737] vdf: detected capacity change from 0 to 4294967296 [ 3.269516] libphy: Fixed MDIO Bus: probed [ 3.276376] usbcore: registered new interface driver usbserial_generic [ 3.278439] usbserial: USB Serial support registered for generic [ 3.280937] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.284599] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.286348] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.288836] mousedev: PS/2 mouse device common for all mice [ 3.291737] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.292281] rtc_cmos 00:05: RTC can wake from S4 [ 3.300344] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.301667] rtc_cmos 00:05: registered as rtc0 [ 3.304725] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.304979] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.308993] intel_pstate: CPU model not supported [ 3.311025] hid: raw HID events driver (C) Jiri Kosina [ 3.312709] usbcore: registered new interface driver usbhid [ 3.313900] usbhid: USB HID core driver [ 3.314815] drop_monitor: Initializing network drop monitor service [ 3.316344] Initializing XFRM netlink socket [ 3.317634] NET: Registered protocol family 10 [ 3.319468] Segment Routing with IPv6 [ 3.320354] NET: Registered protocol family 17 [ 3.321502] mpls_gso: MPLS GSO support [ 3.325616] RAS: Correctable Errors collector initialized. [ 3.326985] AVX version of gcm_enc/dec engaged. [ 3.327991] AES CTR mode by8 optimization enabled [ 3.385579] sched_clock: Marking stable (3385505693, 0)->(4267117535, -881611842) [ 3.388147] registered taskstats version 1 [ 3.390578] Loading compiled-in X.509 certificates [ 3.392857] zswap: loaded using pool lzo/zbud [ 3.418555] Key type big_key registered [ 3.431859] Key type encrypted registered [ 3.433532] ima: No TPM chip found, activating TPM-bypass! [ 3.435451] ima: Allocated hash algorithm: sha1 [ 3.437686] ima: No architecture policies found [ 3.439922] evm: Initialising EVM extended attributes: [ 3.441877] evm: security.selinux [ 3.443152] evm: security.ima [ 3.444340] evm: security.capability [ 3.445743] evm: HMAC attrs: 0x1 [ 3.448216] rtc_cmos 00:05: setting system clock to 2026-09-30 14:48:00 UTC (1790779680) [ 3.454712] debug: unmapping init [mem 0xffffffffa3a03000-0xffffffffa3bfffff] [ 3.458061] debug: unmapping init [mem 0xffffffffa2782000-0xffffffffa2a58fff] [ 3.468117] Write protecting the kernel read-only data: 28672k [ 3.471713] debug: unmapping init [mem 0xffffffffa0e03000-0xffffffffa0ffffff] [ 3.474220] debug: unmapping init [mem 0xffffffffa1714000-0xffffffffa17fffff] [ 3.511463] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.521163] systemd[1]: Detected virtualization kvm. [ 3.523403] systemd[1]: Detected architecture x86-64. [ 3.525428] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.551329] systemd[1]: No hostname configured. [ 3.553237] systemd[1]: Set hostname to . [ 3.557907] random: systemd: uninitialized urandom read (16 bytes read) [ 3.560562] systemd[1]: Initializing machine ID from random generator. [ 3.601465] random: ln: uninitialized urandom read (6 bytes read) [ 3.686671] random: systemd: uninitialized urandom read (16 bytes read) [ 3.689445] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.693599] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.700313] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.292746] device-mapper: uevent: version 1.0.3 [ 4.296115] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.980027] virtio_net virtio0 ens2: renamed from eth0 [ 5.030414] random: fast init done [ 5.063231] scsi host0: ata_piix [ 5.084212] scsi host1: ata_piix [ 5.085396] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.087220] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.641266] dracut-initqueue[577]: RTNETLINK answers: File exists [ 9.950883] random: crng init done [ 9.952477] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.313338] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.480244] printk: systemd: 23 output lines suppressed due to ratelimiting [ 11.656546] SELinux: Disabled at runtime. [ 11.720453] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.729546] systemd[1]: Detected virtualization kvm. [ 11.732124] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.206776] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.209922] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.213810] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.216961] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.219807] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.227649] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.243067] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ 12.410219] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.640490] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.981223] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.993993] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.129799] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.138687] EDAC sbridge: Ver: 1.1.2 [ 14.702210] Key type dns_resolver registered [ 15.050237] NFS: Registering the id_resolver key type [ 15.052474] Key type id_resolver registered [ 15.053980] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg126-server login: [ 66.371508] libcfs: loading out-of-tree module taints kernel. [ 66.420871] Key type ._llcrypt registered [ 66.422976] Key type .llcrypt registered [ 66.525156] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 86.760307] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 88.340390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 88.352842] alg: No test for adler32 (adler32-zlib) [ 90.022966] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 91.267049] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 93.144532] Key type lgssc registered [ 95.472151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 101.238422] hrtimer: interrupt took 2943609 ns [ 117.706704] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 170.789275] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 187.712085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 187.775550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.125406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 189.179692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 189.293956] Lustre: lustre-MDT0000: new disk, initializing [ 189.404727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 189.429348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 194.557716] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.187705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 210.307727] Lustre: 6504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 210.333935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 210.339235] Lustre: Skipped 1 previous similar message [ 210.474345] Lustre: lustre-MDT0001: new disk, initializing [ 210.586849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 210.620269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 210.633699] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 216.540590] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.596178] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 233.275789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 233.774891] Lustre: lustre-OST0000: new disk, initializing [ 233.779297] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 233.796077] Lustre: 8445:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 233.875352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 241.206806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 241.223209] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 241.405039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 241.478799] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 261.210891] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 261.564081] Lustre: lustre-OST0001: new disk, initializing [ 261.590473] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 261.606195] Lustre: 9517:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 261.742522] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 267.882648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 267.911713] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 268.094663] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 270.799727] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 285.275981] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 294.374322] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 303.002889] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing check_logdir /tmp/testlogs/ [ 308.668626] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing yml_node [ 314.899858] Lustre: DEBUG MARKER: Client: 2.17.59.2 [ 317.470363] Lustre: DEBUG MARKER: MDS: 2.17.59.2 [ 320.652702] Lustre: DEBUG MARKER: OSS: 2.17.59.2 [ 322.603209] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 30 10:53:18 EDT 2026 [ 342.038146] Lustre: DEBUG MARKER: only running tests: 33-75 [ 343.975941] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 345.756978] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 364.515489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 364.535920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.555778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 365.540512] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.540770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 365.586097] Lustre: Skipped 1 previous similar message [ 369.252881] Lustre: server umount lustre-MDT0000 complete [ 374.498611] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780051 with bad export cookie 12431527093955789053 [ 374.500595] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 374.508547] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 375.776996] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 375.778985] LustreError: 9742:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 375.810241] Lustre: Skipped 2 previous similar messages [ 375.810774] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 375.855952] LustreError: 9742:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 379.810609] LustreError: 6514:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 379.811152] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 379.825961] LustreError: 6514:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 379.853552] Lustre: Skipped 2 previous similar messages [ 381.364533] Lustre: server umount lustre-MDT0001 complete [ 386.818794] Lustre: server umount lustre-OST0000 complete [ 391.655623] Lustre: server umount lustre-OST0001 complete [ 399.345436] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 410.203997] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 450.150507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 450.431197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 450.484905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 450.629291] Lustre: lustre-MDT0000: new disk, initializing [ 450.760844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.810479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 455.629314] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 468.138072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 468.328225] Lustre: 15124:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 468.401818] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 468.406069] Lustre: Skipped 1 previous similar message [ 468.528829] Lustre: lustre-MDT0001: new disk, initializing [ 468.625383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 468.693186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 468.711743] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 474.732625] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 486.273716] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 493.221475] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 501.579751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 502.166864] Lustre: lustre-OST0000: new disk, initializing [ 502.171442] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 502.184247] Lustre: 17091:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 502.332627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 504.264182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 504.281908] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 504.457934] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 509.589552] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 520.808042] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 526.908118] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 527.221377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 534.256047] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 534.609948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 540.147175] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 540.159628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 540.165735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 540.180019] Lustre: Skipped 2 previous similar messages [ 540.202547] Lustre: Skipped 1 previous similar message [ 543.898884] Lustre: server umount lustre-OST0000 complete [ 550.372094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 550.374978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 550.403114] Lustre: Skipped 1 previous similar message [ 550.429570] Lustre: Skipped 1 previous similar message [ 554.542764] Lustre: server umount lustre-MDT0000 complete [ 555.489366] LustreError: 15135:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 555.517844] LustreError: 15135:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 558.855471] LustreError: 15115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780235 with bad export cookie 12431527093955790593 [ 558.866770] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 558.875403] LustreError: 15115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 559.329561] Lustre: server umount lustre-MDT0001 complete [ 564.460373] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:57:19 (1790780239) [ 567.281502] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 568.689364] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:57:24 (1790780244) [ 577.107783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 577.762656] LustreError: 19500:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.837159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 583.151935] LustreError: 19501:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.929456] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 592.357501] LustreError: 19500:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 592.380024] LustreError: 19500:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 593.879046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 600.631905] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 609.541773] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 615.780297] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 623.089424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 631.125502] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 639.726463] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 662.498625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 662.512477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 662.513146] Lustre: Skipped 1 previous similar message [ 662.531338] Lustre: Skipped 1 previous similar message [ 667.148939] Lustre: server umount lustre-OST0000 complete [ 678.863554] Lustre: server umount lustre-MDT0000 complete [ 681.970440] LustreError: 22005:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 682.007596] LustreError: 22005:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 683.059896] LustreError: 19480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780360 with bad export cookie 12431527093955791475 [ 683.069306] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 683.079241] LustreError: 19480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 683.567284] Lustre: server umount lustre-MDT0001 complete [ 695.184577] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 698.663465] Key type lgssc unregistered [ 698.977306] LNet: 23060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.986903] LNetError: 23060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 700.010895] LNet: Removed LNI 192.168.201.126@tcp [ 701.151245] Key type .llcrypt unregistered [ 701.153337] Key type ._llcrypt unregistered [ 724.998386] Key type ._llcrypt registered [ 725.000798] Key type .llcrypt registered [ 725.114780] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 726.765048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 726.869645] alg: No test for adler32 (adler32-zlib) [ 727.949493] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 728.140466] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 729.808271] Key type lgssc registered [ 731.158517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 739.611736] loop: module loaded [ 739.715391] loop0: detected capacity change from 0 to 204800000 [ 739.951507] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 739.983112] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 740.333524] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 747.581569] loop0: detected capacity change from 0 to 204800000 [ 747.793661] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 747.818605] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 747.828769] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 753.154903] loop0: detected capacity change from 0 to 204800000 [ 759.689938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 759.756973] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 760.977954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 761.047101] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 761.130598] Lustre: lustre-MDT0000: new disk, initializing [ 761.205883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 761.221761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 765.751698] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 774.916244] loop1: detected capacity change from 0 to 204800000 [ 780.376047] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 780.797714] Lustre: lustre-OST07c6: new disk, initializing [ 780.806868] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 780.820215] Lustre: Skipped 1 previous similar message [ 780.825853] Lustre: 26261:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 780.970473] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 787.586641] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 788.600066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 788.610251] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 788.640609] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 793.385349] Lustre: 25248:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 793.395969] Lustre: 25248:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 793.405338] Lustre: 25248:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 793.413302] Lustre: 25248:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 793.420178] Lustre: 25248:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 793.426631] Lustre: 25248:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 803.518594] Lustre: 25249:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 803.535530] Lustre: 25249:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 803.547151] Lustre: 25249:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 803.571131] Lustre: 25249:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 803.583588] Lustre: 25249:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 803.590310] Lustre: 25249:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 808.797324] Lustre: 26793:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 808.811897] Lustre: 26793:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 808.822670] Lustre: 26793:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 808.833086] Lustre: 26793:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.843149] Lustre: 26793:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 808.852903] Lustre: 26793:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.864450] Lustre: 26793:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 808.875913] Lustre: 26793:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.881254] Lustre: 26793:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 808.888433] Lustre: 26793:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.900191] Lustre: 26793:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 808.910873] Lustre: 26793:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.401671] Lustre: 25248:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 813.409851] Lustre: 25248:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 813.415499] Lustre: 25248:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 813.422571] Lustre: 25248:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.429212] Lustre: 25248:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 813.441337] Lustre: 25248:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.448377] Lustre: 25248:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 813.455363] Lustre: 25248:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.463246] Lustre: 25248:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 813.474874] Lustre: 25248:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.490919] Lustre: 25248:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 813.506380] Lustre: 25248:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 821.572871] Lustre: 25248:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 821.581160] Lustre: 25248:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 3 previous similar messages [ 821.590617] Lustre: 25248:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 821.599491] Lustre: 25248:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 821.612282] Lustre: 25248:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 821.621282] Lustre: 25248:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 821.627142] Lustre: 25248:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 821.636262] Lustre: 25248:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 821.646123] Lustre: 25248:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 821.653521] Lustre: 25248:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 821.676931] Lustre: 25248:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 821.685816] Lustre: 25248:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 832.886094] Lustre: 26793:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 832.903760] Lustre: 26793:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 5 previous similar messages [ 832.918598] Lustre: 26793:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 832.928916] Lustre: 26793:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 832.941556] Lustre: 26793:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 832.959596] Lustre: 26793:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 832.968854] Lustre: 26793:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 832.979118] Lustre: 26793:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 832.985856] Lustre: 26793:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 833.000398] Lustre: 26793:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 833.011460] Lustre: 26793:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 833.019884] Lustre: 26793:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 880.611234] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 880.626126] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.666471] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 885.733563] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 890.850491] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 892.384901] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 892.547059] Lustre: server umount lustre-OST07c6 complete [ 903.221836] Lustre: server umount lustre-MDT0000 complete [ 920.809821] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 923.935226] Key type lgssc unregistered [ 924.176976] LNet: 28833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.185193] LNetError: 28833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 924.199161] LNet: Removed LNI 192.168.201.126@tcp [ 925.119086] Key type .llcrypt unregistered [ 925.121746] Key type ._llcrypt unregistered [ 946.928195] Key type ._llcrypt registered [ 946.930759] Key type .llcrypt registered [ 947.024595] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 948.106483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 948.169028] alg: No test for adler32 (adler32-zlib) [ 949.516807] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 949.880180] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 951.625310] Key type lgssc registered [ 953.297858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 962.018262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 962.030826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 963.385394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 970.136403] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 973.801872] LustreError: 30121:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 973.861068] LustreError: 30121:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 978.918506] LustreError: 30122:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 980.584808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 981.239539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 986.352680] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 995.465794] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1001.477276] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1008.515273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1009.085189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1015.266565] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1022.754522] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1033.650775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1052.134880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1052.145246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1052.171753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1054.185884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1054.202170] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1054.206714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1057.778962] Lustre: server umount lustre-OST0000 complete [ 1063.394096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1063.405145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1063.436992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1063.439272] Lustre: Skipped 1 previous similar message [ 1068.548113] Lustre: server umount lustre-MDT0000 complete [ 1069.546648] LustreError: 31118:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1069.569229] LustreError: 31118:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1072.645665] LustreError: 30100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780749 with bad export cookie 10267893384116953138 [ 1072.650177] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1072.653259] LustreError: 30100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1072.862791] Lustre: server umount lustre-MDT0001 complete [ 1084.616853] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 1087.248775] Key type lgssc unregistered [ 1087.533187] LNet: 33881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1087.543945] LNetError: 33881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1088.615308] LNet: Removed LNI 192.168.201.126@tcp [ 1089.735095] Key type .llcrypt unregistered [ 1089.738086] Key type ._llcrypt unregistered [ 1109.143736] Key type ._llcrypt registered [ 1109.145116] Key type .llcrypt registered [ 1109.217588] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 1110.623639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1110.682171] alg: No test for adler32 (adler32-zlib) [ 1111.885569] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 1112.183299] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 1113.824340] Key type lgssc registered [ 1114.943125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1122.669238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1122.701362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1124.313199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1128.771044] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1134.564858] LustreError: 35169:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1134.602643] LustreError: 35169:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1137.463618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1137.958621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1142.507541] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1150.982926] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1156.693358] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1163.508282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1163.740535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1169.917552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1170.224378] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1179.610685] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1192.426044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1192.431457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1192.450632] Lustre: Skipped 1 previous similar message [ 1192.452198] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1192.478843] Lustre: Skipped 1 previous similar message [ 1197.549370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1197.555099] Lustre: Skipped 1 previous similar message [ 1198.676693] Lustre: server umount lustre-OST0000 complete [ 1204.710593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1204.726595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1204.767046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1207.781496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1207.783873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1209.222650] Lustre: server umount lustre-MDT0000 complete [ 1212.924953] LustreError: 36185:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1212.946598] LustreError: 36185:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1213.125107] LustreError: 35904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780890 with bad export cookie 2612551731045941425 [ 1213.131411] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1213.160543] LustreError: 35904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1213.677254] Lustre: server umount lustre-MDT0001 complete [ 1225.500318] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 1228.168638] Key type lgssc unregistered [ 1228.397253] LNet: 38724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1228.403638] LNetError: 38724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1229.495219] LNet: Removed LNI 192.168.201.126@tcp [ 1230.204391] Key type .llcrypt unregistered [ 1230.206137] Key type ._llcrypt unregistered [ 1252.855857] Key type ._llcrypt registered [ 1252.860484] Key type .llcrypt registered [ 1252.941313] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 1254.402361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1254.660965] alg: No test for adler32 (adler32-zlib) [ 1255.647777] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 1255.827163] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 1257.472216] Key type lgssc registered [ 1258.492378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1266.029901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1266.068552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1267.408887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1271.593616] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1277.925942] LustreError: 40155:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1277.947469] LustreError: 40155:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1280.781598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.094673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1285.594965] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1293.909788] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1300.876870] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1308.087346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1308.511692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1312.622404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1315.127537] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1323.228102] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1336.255785] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.26@tcp (stopping) [ 1337.314516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1337.325839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1337.334465] Lustre: Skipped 2 previous similar messages [ 1337.335339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1337.359118] Lustre: Skipped 2 previous similar messages [ 1340.031132] Lustre: server umount lustre-MDT0000 complete [ 1341.339950] LustreError: 40151:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1342.442038] LustreError: 40149:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1342.460404] LustreError: 40149:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1344.057300] LustreError: 40135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781021 with bad export cookie 418501872435215141 [ 1344.069161] LustreError: 40135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1344.069494] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1346.458276] LustreError: 42647:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1346.489829] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.26@tcp (stopping) [ 1347.554406] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1349.602109] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1349.606211] Lustre: Skipped 1 previous similar message [ 1350.476274] Lustre: server umount lustre-MDT0001 complete [ 1366.718304] Lustre: server umount lustre-OST0000 complete [ 1385.396905] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 1388.315923] Key type lgssc unregistered [ 1388.562680] LNet: 44013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1388.568710] LNetError: 44013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.593691] LNet: Removed LNI 192.168.201.126@tcp [ 1389.564845] Key type .llcrypt unregistered [ 1389.569351] Key type ._llcrypt unregistered [ 1417.100966] Key type ._llcrypt registered [ 1417.104957] Key type .llcrypt registered [ 1417.239791] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 1418.683338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1418.726718] alg: No test for adler32 (adler32-zlib) [ 1420.009777] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 1420.358450] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 1422.024221] Key type lgssc registered [ 1423.551705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1432.492593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1432.524172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1434.167430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1439.261104] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1444.324783] LustreError: 45446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1444.345487] LustreError: 45446:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1449.371051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1449.441903] LustreError: 45447:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1449.633537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1454.802978] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1464.920645] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1471.779734] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1480.169730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1480.578871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1487.133581] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1494.917522] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1504.182188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1512.386447] Lustre: lustre-OST0000: Not available for connect from 192.168.201.26@tcp (stopping) [ 1514.468183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1514.475652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1514.496227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1514.501258] Lustre: Skipped 1 previous similar message [ 1517.483212] Lustre: lustre-OST0000: Not available for connect from 192.168.201.26@tcp (stopping) [ 1518.475049] Lustre: server umount lustre-OST0000 complete [ 1540.065222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1540.069296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1540.073974] Lustre: Skipped 2 previous similar messages [ 1540.085492] Lustre: Skipped 1 previous similar message [ 1543.770370] Lustre: server umount lustre-MDT0000 complete [ 1545.186699] LustreError: 45446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1545.217339] LustreError: 45446:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1547.681053] LustreError: 45427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781224 with bad export cookie 17795083076960172818 [ 1547.688239] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1547.692422] LustreError: 45427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1554.059350] Lustre: server umount lustre-MDT0001 complete [ 1566.937168] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 1569.999879] Key type lgssc unregistered [ 1570.297649] LNet: 49177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1570.307079] LNetError: 49177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1570.340816] LNet: Removed LNI 192.168.201.126@tcp [ 1571.226581] Key type .llcrypt unregistered [ 1571.229384] Key type ._llcrypt unregistered [ 1595.996174] Key type ._llcrypt registered [ 1595.998750] Key type .llcrypt registered [ 1596.107516] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 1597.270649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1597.332403] alg: No test for adler32 (adler32-zlib) [ 1598.443828] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 1598.787603] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 1600.522698] Key type lgssc registered [ 1601.930507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1610.362292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1610.401402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1611.950798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1616.420034] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1621.478057] LustreError: 50611:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1621.513660] LustreError: 50611:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1625.577511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1626.018608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1631.416449] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1639.480687] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1645.103543] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1651.967745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1652.219568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1656.329444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1658.680677] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1666.801430] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1676.660430] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1681.381299] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1694.184447] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-3011h16m08s [ 1695.904831] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1697.762052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1697.776377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1697.795425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1698.854470] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1698.856848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1698.896869] Lustre: Skipped 2 previous similar messages [ 1699.938314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1699.944781] Lustre: Skipped 1 previous similar message [ 1702.052131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1702.069252] Lustre: Skipped 4 previous similar messages [ 1706.165529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1706.173222] Lustre: Skipped 5 previous similar messages [ 1712.097965] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1712.301655] LustreError: 53311:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1712.357948] Lustre: server umount lustre-MDT0000 complete [ 1713.386414] LustreError: 50610:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1713.420034] LustreError: 50610:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1715.429837] LustreError: 50611:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1715.442665] LustreError: 50611:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1716.359346] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1719.589991] LustreError: 50611:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1719.621320] LustreError: 50611:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 1723.751575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1723.832435] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1724.082576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1724.139254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1725.116922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1728.976926] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1732.489729] Lustre: DEBUG MARKER: Wait for df (36968) ... [ 1734.016603] Lustre: DEBUG MARKER: done [ 1739.746191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1739.755061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1739.767135] Lustre: Skipped 1 previous similar message [ 1739.775629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1739.783284] Lustre: Skipped 8 previous similar messages [ 1753.056150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1753.263602] Lustre: server umount lustre-OST0000 complete [ 1761.250401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1761.254228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1761.278930] Lustre: Skipped 1 previous similar message [ 1761.282603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1761.309943] Lustre: Skipped 7 previous similar messages [ 1764.420278] Lustre: server umount lustre-MDT0000 complete [ 1768.449604] LustreError: 50605:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1768.468548] LustreError: 50605:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 1768.994340] LustreError: 50590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781446 with bad export cookie 2062579465589205740 [ 1769.001242] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1769.005174] LustreError: 50590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1769.263961] Lustre: server umount lustre-MDT0001 complete [ 1782.520768] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 1785.302872] Key type lgssc unregistered [ 1785.641829] LNet: 55535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1785.645691] LNetError: 55535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1785.666443] LNet: Removed LNI 192.168.201.126@tcp [ 1786.526170] Key type .llcrypt unregistered [ 1786.528430] Key type ._llcrypt unregistered [ 1827.130174] Key type ._llcrypt registered [ 1827.137175] Key type .llcrypt registered [ 1827.291478] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 1828.289546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1828.367939] alg: No test for adler32 (adler32-zlib) [ 1829.580834] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 1829.838912] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 1831.508899] Key type lgssc registered [ 1832.731975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1840.971223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1840.995839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1842.337464] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1842.369484] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1842.562059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1846.889537] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1854.979941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1855.127077] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1855.150913] Lustre: 57757:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1855.420743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1859.389530] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1866.961612] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1872.942787] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1880.884105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1881.035106] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1881.259229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1888.129074] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1889.630532] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1897.153762] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1905.537590] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-3011h19m41s [ 1907.344513] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1914.923905] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1917.477144] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1917.560626] LustreError: 59217:0:(mdt_handler.c:2345:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1937.881855] Lustre: lustre-MDT0000: Client b35e33e8-8119-49b9-a7f9-db0b2323fb58 (at 192.168.201.26@tcp) reconnecting [ 1957.632563] LustreError: 59217:0:(mdt_handler.c:2345:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1961.208279] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1963.952932] Lustre: DEBUG MARKER: done [ 1971.685254] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.700124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1971.702075] Lustre: Skipped 1 previous similar message [ 1971.710376] Lustre: Skipped 1 previous similar message [ 1971.711189] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1975.884854] Lustre: server umount lustre-OST0000 complete [ 1981.923586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1981.924894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1981.936591] Lustre: Skipped 1 previous similar message [ 1981.945920] Lustre: Skipped 1 previous similar message [ 1983.014872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1983.027390] Lustre: Skipped 1 previous similar message [ 1985.881423] Lustre: server umount lustre-MDT0000 complete [ 1989.567416] LustreError: 57750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781666 with bad export cookie 965211694800715495 [ 1989.579074] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1989.592289] LustreError: 57750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1990.068285] Lustre: server umount lustre-MDT0001 complete [ 2002.596446] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2005.750836] Key type lgssc unregistered [ 2006.100532] LNet: 62089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2006.110587] LNetError: 62089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2006.131523] LNet: Removed LNI 192.168.201.126@tcp [ 2006.840821] Key type .llcrypt unregistered [ 2006.842802] Key type ._llcrypt unregistered [ 2049.656835] Key type ._llcrypt registered [ 2049.658899] Key type .llcrypt registered [ 2049.751390] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 2050.719704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2051.020433] alg: No test for adler32 (adler32-zlib) [ 2052.159628] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 2052.394768] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 2054.056217] Key type lgssc registered [ 2055.010590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2057.777670] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2057.808784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2059.176901] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2059.202342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2059.392929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2060.298648] Lustre: Failing over lustre-MDT0000 [ 2060.503279] Lustre: server umount lustre-MDT0000 complete [ 2067.269796] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 11:22:22 (1790781742) [ 2073.678232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.989026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2077.749329] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2086.041663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2086.173925] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2086.200712] Lustre: 64900:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2086.497681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2090.756209] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2098.754957] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2105.292990] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2112.930378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2113.128193] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2113.293230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2119.916564] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2121.904885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2127.942651] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2134.443095] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2142.695792] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2142.698195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2142.706101] Lustre: Skipped 1 previous similar message [ 2142.719995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2147.037617] Lustre: server umount lustre-MDT0000 complete [ 2150.670511] LustreError: 64892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781827 with bad export cookie 2494569323438348100 [ 2150.676691] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2150.690581] LustreError: 64892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2151.102941] Lustre: server umount lustre-MDT0001 complete [ 2154.933831] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2160.072463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2168.290460] Lustre: 63814:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790781829/real 1790781829] req@000000005ab2ac7a x1877770742309248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790781845 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2168.364461] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.393991] Lustre: Skipped 1 previous similar message [ 2170.842944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2171.488274] Lustre: 63814:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790781832/real 1790781832] req@00000000a448a4de x1877770742309376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790781848 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2174.432217] Lustre: 63816:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790781835/real 1790781835] req@00000000becf460e x1877770742309504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790781851 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2175.585376] LustreError: 63812:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f2cbf97e x1877770742310272/t0(0) o250->MGC192.168.201.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2175.909028] LustreError: 66749:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2175.922656] LustreError: 66749:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2175.957240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2176.047826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2176.928302] Lustre: 63816:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790781837/real 1790781837] req@00000000610c0da4 x1877770742309760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790781853 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2177.959904] LustreError: 66750:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2179.041659] LustreError: 68672:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2179.057557] LustreError: 68672:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2181.093442] Lustre: 63814:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790781842/real 1790781842] req@000000003c79acbe x1877770742309888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790781858 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2181.264698] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2184.167846] LustreError: 66749:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2184.185338] LustreError: 66749:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2189.294928] LustreError: 68807:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2189.329873] LustreError: 68807:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2192.616412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.069120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2197.166829] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2199.480670] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2206.915146] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2213.190820] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2220.091479] Lustre: server umount lustre-MDT0000 complete [ 2223.077068] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2223.078495] LustreError: 66749:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2223.099562] Lustre: Skipped 1 previous similar message [ 2223.115122] LustreError: 66749:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2223.693811] LustreError: 64894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781900 with bad export cookie 2494569323438350375 [ 2223.697884] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2223.702389] LustreError: 64894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2224.020237] Lustre: server umount lustre-MDT0001 complete [ 2229.538634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2238.292731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2244.576174] Lustre: 63814:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790781905/real 1790781905] req@00000000fcc72c16 x1877770742338432/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790781921 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2244.601884] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2244.610259] Lustre: Skipped 1 previous similar message [ 2248.737024] LustreError: 63812:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000be708e24 x1877770742339328/t0(0) o250->MGC192.168.201.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2249.070186] LustreError: 66749:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2249.154640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2249.209931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2253.823076] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2262.714155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2264.182773] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2264.197526] Lustre: Skipped 1 previous similar message [ 2266.863913] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2274.213670] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2280.091214] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2286.361131] Lustre: server umount lustre-MDT0000 complete [ 2288.619049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2288.636949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2288.653438] LustreError: 66749:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2288.681707] LustreError: 66749:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2289.838468] LustreError: 64894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781966 with bad export cookie 2494569323438351754 [ 2289.846497] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2289.849190] LustreError: 64894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2290.162950] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2290.169387] Lustre: Skipped 2 previous similar messages [ 2292.222047] Lustre: server umount lustre-MDT0001 complete [ 2298.407886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2301.734752] Lustre: DEBUG MARKER: files compared the same [ 2303.996653] Lustre: server umount lustre-OST0000 complete [ 2319.840721] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2323.370331] Key type lgssc unregistered [ 2323.922864] LNet: 74625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2323.940139] LNetError: 74625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2323.977732] LNet: Removed LNI 192.168.201.126@tcp [ 2324.986645] Key type .llcrypt unregistered [ 2324.988631] Key type ._llcrypt unregistered [ 2351.640486] Key type ._llcrypt registered [ 2351.643749] Key type .llcrypt registered [ 2351.799523] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 2353.493923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2353.520027] alg: No test for adler32 (adler32-zlib) [ 2354.738644] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 2355.209160] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 2356.993343] Key type lgssc registered [ 2358.795375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2369.434484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2369.507648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2371.227786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2376.951986] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug +malloc all [ 2381.794440] LustreError: 76063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.817191] LustreError: 76063:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2386.189619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2386.480942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2391.458533] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug +malloc all [ 2399.293795] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2405.835956] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2413.028977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2413.457681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2416.562348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2419.609469] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug +malloc all [ 2427.260338] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2442.215715] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2442.216356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2442.227352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2442.243976] Lustre: Skipped 1 previous similar message [ 2444.607613] Lustre: server umount lustre-OST0000 complete [ 2452.449181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2452.453513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2452.469121] Lustre: Skipped 1 previous similar message [ 2452.470860] Lustre: Skipped 1 previous similar message [ 2454.419478] Lustre: server umount lustre-MDT0000 complete [ 2457.572199] LustreError: 76062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2457.603660] LustreError: 76062:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2457.626125] LustreError: 76044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782134 with bad export cookie 15736533260932257347 [ 2457.631218] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2457.633840] LustreError: 76044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2457.871250] Lustre: server umount lustre-MDT0001 complete [ 2470.963830] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2474.065355] Key type lgssc unregistered [ 2474.499592] LNet: 79595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2474.510849] LNetError: 79595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2474.557269] LNet: Removed LNI 192.168.201.126@tcp [ 2475.635519] Key type .llcrypt unregistered [ 2475.639771] Key type ._llcrypt unregistered [ 2502.334190] Key type ._llcrypt registered [ 2502.340200] Key type .llcrypt registered [ 2502.460796] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 2503.749981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2503.889397] alg: No test for adler32 (adler32-zlib) [ 2505.092645] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 2505.323414] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 2506.984888] Key type lgssc registered [ 2508.423773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2515.959760] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2515.977776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2531.552434] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2536.672288] LustreError: 81006:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.126@tcp: failed processing log, type 4: rc = -110 [ 2563.489070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2569.865487] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2577.609267] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2586.026207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2586.151894] Lustre: *** cfs_fail_loc=706, val=0*** [ 2586.617248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2586.701141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2591.321705] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2591.718570] LustreError: 81031:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2591.731171] LustreError: 81031:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2596.834578] LustreError: 81032:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.901212] LustreError: 82092:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.931869] LustreError: 82092:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2600.116209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2600.661690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2606.113450] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2615.602631] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2622.681265] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2626.018655] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2626.039704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2626.050930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2626.539883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2626.567144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2631.516777] Lustre: server umount lustre-OST0000 complete [ 2636.261979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2636.271382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2636.282135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2641.638979] Lustre: server umount lustre-MDT0000 complete [ 2641.890041] LustreError: 82097:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.908387] LustreError: 82097:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2645.999818] LustreError: 82092:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2646.022302] LustreError: 82092:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2646.179979] LustreError: 81014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782323 with bad export cookie 2754586280374073911 [ 2646.192846] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2646.201609] LustreError: 81014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2646.813375] Lustre: server umount lustre-MDT0001 complete [ 2659.716706] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2662.439344] Key type lgssc unregistered [ 2662.654751] LNet: 84622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2662.664210] LNetError: 84622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2662.690582] LNet: Removed LNI 192.168.201.126@tcp [ 2663.378908] Key type .llcrypt unregistered [ 2663.381387] Key type ._llcrypt unregistered [ 2689.720889] Key type ._llcrypt registered [ 2689.725570] Key type .llcrypt registered [ 2689.884505] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 2691.116078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2691.178437] alg: No test for adler32 (adler32-zlib) [ 2692.251883] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 2692.566676] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 2694.256233] Key type lgssc registered [ 2695.439205] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2702.723519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2702.740710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2708.646141] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2716.558495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2716.947218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2721.554380] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2722.276311] LustreError: 86740:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2722.301825] LustreError: 86740:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2726.376199] LustreError: 86743:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.400048] LustreError: 86743:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2730.368161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2730.662974] LustreError: 86740:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2730.684937] LustreError: 86740:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2730.703466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2736.108271] LustreError: 86739:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2736.135696] LustreError: 86739:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2737.586456] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2741.237748] LustreError: 86744:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2741.259965] LustreError: 86744:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2746.757550] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2746.777610] Lustre: MGS: Client faf56257-a3ce-45e8-8fbf-a7518a44b505 (at 0@lo) reconnecting [ 2746.795615] LustreError: 86754:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2746.983459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2747.064355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2751.110761] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2767.348506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2767.357281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2767.360436] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2767.380647] Lustre: Skipped 1 previous similar message [ 2767.424244] Lustre: Skipped 1 previous similar message [ 2772.407978] Lustre: server umount lustre-OST0000 complete [ 2782.689539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2782.700211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2782.702462] Lustre: Skipped 1 previous similar message [ 2782.719986] Lustre: Skipped 1 previous similar message [ 2784.001151] Lustre: server umount lustre-MDT0000 complete [ 2787.824909] LustreError: 86743:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2787.841022] LustreError: 86743:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2788.461457] Lustre: server umount lustre-MDT0001 complete [ 2793.821957] Lustre: server umount MGS complete [ 2808.471838] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 11:34:44 (1790782484) [ 2843.968308] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 2855.719681] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 2895.021529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2900.397354] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2911.692787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2912.110661] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2912.180558] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2912.280732] Lustre: lustre-MDT0001: new disk, initializing [ 2912.378452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2912.428558] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2912.449557] Lustre: 93930:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2912.466228] LustreError: 93930:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2917.259219] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2927.616519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2927.926598] Lustre: lustre-OST0000: new disk, initializing [ 2927.930828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2927.936856] Lustre: 94699:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2928.038689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2935.207125] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2938.980693] Lustre: 95155:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000005abb1b75 x1877771413398144/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790782632 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2944.789478] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2951.913542] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2951.938201] Lustre: MGS: Client b6743baf-8968-4ff7-acfd-afc75f4232bf (at 0@lo) reconnecting [ 2951.994822] Lustre: 93198:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2952.035501] LustreError: 93198:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2952.040867] LustreError: 93198:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2952.072430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2952.173046] Lustre: lustre-MDT0000: new disk, initializing [ 2952.357984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2952.383488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2952.536919] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2953.442271] Lustre: 95767:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000009892a6fa x1877771413404800/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1790782646 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2957.157662] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2959.330561] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2962.604654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2962.616574] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2962.722143] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2988.517082] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2988.537050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2988.563727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2995.173859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2995.184393] Lustre: Skipped 3 previous similar messages [ 3000.293349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3000.301489] Lustre: Skipped 1 previous similar message [ 3002.336299] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3002.577279] Lustre: server umount lustre-OST0000 complete [ 3010.529259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3010.535057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3010.541000] Lustre: Skipped 2 previous similar messages [ 3010.543490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3022.304209] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3022.473385] Lustre: server umount lustre-MDT0000 complete [ 3025.890278] LustreError: 93906:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3025.898871] LustreError: 93906:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3032.966726] Lustre: server umount lustre-MDT0001 complete [ 3038.854258] Lustre: server umount MGS complete [ 3051.421105] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 11:38:47 (1790782727) [ 3074.517745] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 3078.001467] Key type lgssc unregistered [ 3078.360875] LNet: 98278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3078.366357] LNetError: 98278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3078.390109] LNet: Removed LNI 192.168.201.126@tcp [ 3079.452975] Key type .llcrypt unregistered [ 3079.456579] Key type ._llcrypt unregistered [ 3103.211415] Key type ._llcrypt registered [ 3103.215538] Key type .llcrypt registered [ 3103.329159] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3104.386958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3104.415669] alg: No test for adler32 (adler32-zlib) [ 3105.885651] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 3106.122595] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 3107.872239] Key type lgssc registered [ 3109.199262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3116.954945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3116.986891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3118.499643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3123.119169] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3128.803307] LustreError: 99517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3128.824123] LustreError: 99517:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3131.543700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3131.917022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3136.557446] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3143.369526] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3149.301391] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3156.466982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3156.909221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3162.410854] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3163.071068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3169.913166] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3183.585687] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3183.596708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3183.608589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3186.657528] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3186.659494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3186.669992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3189.336741] Lustre: server umount lustre-OST0000 complete [ 3193.315024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3193.322867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3193.343227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3193.351738] Lustre: Skipped 1 previous similar message [ 3196.898329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3196.922771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3199.598869] Lustre: server umount lustre-MDT0000 complete [ 3202.026135] LustreError: 100953:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3202.061403] LustreError: 100953:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3203.245218] LustreError: 99499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782880 with bad export cookie 7378790453667515978 [ 3203.249241] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3203.264520] LustreError: 99499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3203.775414] Lustre: server umount lustre-MDT0001 complete [ 3217.845038] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 3221.551454] Key type lgssc unregistered [ 3221.943928] LNet: 103072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3221.952418] LNetError: 103072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3221.999121] LNet: Removed LNI 192.168.201.126@tcp [ 3223.176288] Key type .llcrypt unregistered [ 3223.178633] Key type ._llcrypt unregistered [ 3243.602524] Key type ._llcrypt registered [ 3243.604437] Key type .llcrypt registered [ 3243.688850] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3244.996398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3245.018040] alg: No test for adler32 (adler32-zlib) [ 3246.101308] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 3246.345720] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 3248.097591] Key type lgssc registered [ 3249.531121] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3255.340633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3255.348287] LustreError: 104164:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3255.384339] LustreError: 104172:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3255.388248] LustreError: 104164:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4966 [ 3255.392505] LustreError: 104164:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3255.397943] LustreError: 104164:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3255.419556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3257.119881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3265.370041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3265.577328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3269.795194] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3275.533246] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing lsmod [ 3279.033358] LustreError: 105376:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3279.049799] LustreError: 105380:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3279.067890] LustreError: 105376:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4977 [ 3279.081895] LustreError: 105376:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3279.087641] LustreError: 105376:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3279.106563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3279.561871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3282.638768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3282.840374] LustreError: 104230:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 3282.860178] LustreError: 104230:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 3285.986134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3285.999244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3286.019486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3288.039431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3288.041115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3288.061252] Lustre: Skipped 1 previous similar message [ 3288.075921] Lustre: Skipped 1 previous similar message [ 3290.156610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3290.175147] Lustre: Skipped 2 previous similar messages [ 3295.204448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3295.216704] Lustre: Skipped 2 previous similar messages [ 3297.248136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3297.553840] Lustre: server umount lustre-MDT0000 complete [ 3300.328272] LustreError: 104210:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3300.350241] LustreError: 104210:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3302.158380] LustreError: 104190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782979 with bad export cookie 1017680834136932944 [ 3302.180816] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3302.458218] LustreError: 104640:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003448ed38 x1877771994291200/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3302.619681] Lustre: server umount lustre-MDT0001 complete [ 3317.624126] Lustre: server umount lustre-OST0000 complete [ 3327.286331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3327.844654] LustreError: 106444:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3327.949655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3333.089959] LustreError: 106445:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3333.919174] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3337.200449] LustreError: 106444:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3342.317810] LustreError: 106445:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3345.908902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3346.286298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3351.714349] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3360.151940] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3366.423409] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3373.958289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3374.243167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3380.670794] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3390.880107] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3397.821861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3407.329428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3407.342337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3407.365132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3408.359175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3413.456955] Lustre: server umount lustre-OST0000 complete [ 3418.081791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3418.094608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3418.106952] Lustre: Skipped 1 previous similar message [ 3418.114031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3418.121411] Lustre: Skipped 2 previous similar messages [ 3424.613466] Lustre: server umount lustre-MDT0000 complete [ 3428.407929] LustreError: 106426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790783105 with bad export cookie 1017680834136933875 [ 3428.410127] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3428.432682] LustreError: 106426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3428.890866] Lustre: server umount lustre-MDT0001 complete [ 3443.355366] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 3446.754108] Key type lgssc unregistered [ 3447.055909] LNet: 109999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3447.061956] LNetError: 109999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3447.095883] LNet: Removed LNI 192.168.201.126@tcp [ 3448.023286] Key type .llcrypt unregistered [ 3448.027361] Key type ._llcrypt unregistered [ 3476.200513] Key type ._llcrypt registered [ 3476.213481] Key type .llcrypt registered [ 3476.423501] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3477.933407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3478.065992] alg: No test for adler32 (adler32-zlib) [ 3479.196610] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 3479.460770] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 3481.208326] Key type lgssc registered [ 3482.408378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3490.434183] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3490.469163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3492.030498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3496.570205] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3502.562284] LustreError: 111430:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3502.583854] LustreError: 111430:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3504.668973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3504.944310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3509.398901] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3517.276754] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3523.785216] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3531.268692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3531.511533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3537.667847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3537.808928] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3546.278477] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3555.034912] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3560.449394] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3568.614205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3568.627069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3568.651682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3570.663112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3570.670778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3570.674147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3570.677229] Lustre: Skipped 1 previous similar message [ 3575.782311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3575.787812] Lustre: Skipped 1 previous similar message [ 3580.384633] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3580.544399] Lustre: server umount lustre-OST0000 complete [ 3587.052757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3587.068136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3587.108443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3591.660906] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3591.665828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3591.708666] Lustre: Skipped 1 previous similar message [ 3593.324277] Lustre: server umount lustre-MDT0000 complete [ 3596.770804] LustreError: 111431:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3596.801740] LustreError: 111431:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3598.193316] LustreError: 111410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790783275 with bad export cookie 5429456620560481663 [ 3598.197859] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3598.220492] LustreError: 111410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3598.748750] Lustre: server umount lustre-MDT0001 complete [ 3617.906204] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 3621.356791] Key type lgssc unregistered [ 3621.643421] LNet: 115095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3621.649655] LNetError: 115095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3621.663524] LNet: Removed LNI 192.168.201.126@tcp [ 3622.426802] Key type .llcrypt unregistered [ 3622.429588] Key type ._llcrypt unregistered [ 3647.198598] Key type ._llcrypt registered [ 3647.206763] Key type .llcrypt registered [ 3647.401694] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3650.410969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3650.489000] alg: No test for adler32 (adler32-zlib) [ 3651.899174] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 3652.207968] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 3654.016194] Key type lgssc registered [ 3655.725830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3664.647235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3664.684959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3666.291173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3672.368762] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3676.663762] LustreError: 116332:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3676.689770] LustreError: 116332:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3681.764510] LustreError: 116333:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3683.050182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3683.506688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3689.166836] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3698.427695] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3706.490213] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3714.592965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3715.144218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3722.593959] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3732.197305] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3736.634873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3748.836429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3748.843462] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3748.857221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3750.161505] Lustre: server umount lustre-OST0000 complete [ 3754.988819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3754.995080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3755.004976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3757.026845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3757.029213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3762.145477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3762.149741] Lustre: Skipped 2 previous similar messages [ 3764.286875] Lustre: server umount lustre-MDT0000 complete [ 3766.247305] LustreError: 116332:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3766.271939] LustreError: 116332:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3768.935505] LustreError: 116314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790783445 with bad export cookie 13698785107329469007 [ 3768.944438] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3768.948783] LustreError: 116314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3769.286743] Lustre: server umount lustre-MDT0001 complete [ 3782.758143] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 3785.934645] Key type lgssc unregistered [ 3786.279129] LNet: 119886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3786.295568] LNetError: 119886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3786.318610] LNet: Removed LNI 192.168.201.126@tcp [ 3787.313570] Key type .llcrypt unregistered [ 3787.316110] Key type ._llcrypt unregistered [ 3814.948811] Key type ._llcrypt registered [ 3814.950927] Key type .llcrypt registered [ 3815.057625] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3816.162284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3816.309635] alg: No test for adler32 (adler32-zlib) [ 3817.606361] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 3817.923129] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 3819.568205] Key type lgssc registered [ 3821.082462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3829.076865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3829.087119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3830.596301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3834.809779] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3839.975500] LustreError: 121318:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3839.994659] LustreError: 121318:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3843.234894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3843.545896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3847.948757] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3855.165652] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3861.762781] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3869.073852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3869.442148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3875.573210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3875.873506] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3883.804381] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3894.474391] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3898.298467] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3898.302868] Lustre: Skipped 2 previous similar messages [ 3901.089639] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3901.154464] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3902.775653] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3902.780376] Lustre: Skipped 2 previous similar messages [ 3911.246244] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3911.250331] Lustre: Skipped 1 previous similar message [ 3916.092136] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.26@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3916.108581] Lustre: Skipped 2 previous similar messages [ 3916.447171] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.26@tcp 192.168.0.[2,10]@tcp [ 3928.547343] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.26/24@tcp in log lustre-MDT0000 [ 3928.552562] Lustre: Skipped 2 previous similar messages [ 3938.430062] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.26/24@tcp [ 3938.443098] Lustre: Skipped 1 previous similar message [ 3945.083053] Lustre: server umount lustre-OST0000 complete [ 3950.563142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3950.582634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3950.595400] Lustre: Skipped 1 previous similar message [ 3955.683077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3955.693962] Lustre: Skipped 1 previous similar message [ 3958.847688] Lustre: server umount lustre-MDT0000 complete [ 3960.803712] LustreError: 121319:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3960.826391] LustreError: 121319:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3961.827346] Lustre: 120622:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790783622/real 1790783622] req@0000000034a2263b x1877772593073920/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790783638 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3961.860693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3961.881965] Lustre: Skipped 1 previous similar message [ 3961.890485] LustreError: 122341:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3961.912390] LustreError: 122341:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3963.060306] LustreError: 121299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790783640 with bad export cookie 6930642636490723895 [ 3963.067047] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3963.068485] LustreError: 121299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3963.546142] Lustre: server umount lustre-MDT0001 complete [ 3975.937354] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 3979.154399] Key type lgssc unregistered [ 3979.698840] LNet: 126314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3979.718295] LNetError: 126314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3980.784289] LNet: Removed LNI 192.168.201.126@tcp [ 3981.605915] Key type .llcrypt unregistered [ 3981.608929] Key type ._llcrypt unregistered [ 4014.758075] Key type ._llcrypt registered [ 4014.760030] Key type .llcrypt registered [ 4014.872053] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 4016.226861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4016.325924] alg: No test for adler32 (adler32-zlib) [ 4017.327796] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 4017.589932] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 4019.304231] Key type lgssc registered [ 4020.416320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4028.884588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4028.902202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4030.404376] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4030.433258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4030.440801] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.26/24@tcp [ 4034.996341] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4040.683819] LustreError: 127891:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4040.703330] LustreError: 127891:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4043.887523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4044.301228] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4044.337306] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.26/24@tcp [ 4044.392119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4049.264750] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4058.060830] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4064.398569] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4071.122055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4071.577422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4075.720205] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4078.659632] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4087.036446] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4101.609023] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4101.625900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4101.636683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4104.677049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4104.678030] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4104.697581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4106.488767] Lustre: server umount lustre-OST0000 complete [ 4110.817314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4110.827244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4110.842741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4110.854482] Lustre: Skipped 1 previous similar message [ 4114.914733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4114.918285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4116.314908] Lustre: server umount lustre-MDT0000 complete [ 4120.009472] LustreError: 127873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790783797 with bad export cookie 16899492782624941438 [ 4120.019458] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4120.448605] Lustre: server umount lustre-MDT0001 complete [ 4131.523183] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 4133.827587] Key type lgssc unregistered [ 4134.062660] LNet: 131476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4134.072154] LNetError: 131476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4134.088680] LNet: Removed LNI 192.168.201.126@tcp [ 4134.822277] Key type .llcrypt unregistered [ 4134.824317] Key type ._llcrypt unregistered [ 4162.119883] Key type ._llcrypt registered [ 4162.121181] Key type .llcrypt registered [ 4162.191393] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 4178.992578] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 4179.971743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4180.008312] alg: No test for adler32 (adler32-zlib) [ 4181.033299] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 4181.249695] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 4182.929728] Key type lgssc registered [ 4184.227951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4223.449819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4223.486745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4225.091766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4225.127267] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4225.266358] Lustre: lustre-MDT0000: new disk, initializing [ 4225.429458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4225.479355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4231.729970] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4244.166691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4244.355177] Lustre: 134896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4244.453522] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4244.463707] Lustre: Skipped 1 previous similar message [ 4244.546759] Lustre: lustre-MDT0001: new disk, initializing [ 4244.636279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4244.671214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4244.690427] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4249.420704] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4260.009148] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4266.189783] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4273.098269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4273.354460] Lustre: lustre-OST0000: new disk, initializing [ 4273.358876] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4273.367651] Lustre: 136864:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4273.456942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4280.073291] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4283.008425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4283.034854] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4283.226693] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4289.476802] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4293.833141] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4294.014571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4298.522446] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4298.695550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4303.842985] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4303.843668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4303.856515] LustreError: Skipped 1 previous similar message [ 4303.859826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4303.884322] Lustre: Skipped 1 previous similar message [ 4303.904931] Lustre: Skipped 1 previous similar message [ 4306.822401] Lustre: server umount lustre-OST0000 complete [ 4311.525426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4311.540488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4311.559677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4314.081619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4314.083621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4314.100260] Lustre: Skipped 1 previous similar message [ 4319.201912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4319.207019] Lustre: Skipped 1 previous similar message [ 4324.321517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4324.323893] Lustre: Skipped 1 previous similar message [ 4325.344141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4325.703987] Lustre: server umount lustre-MDT0000 complete [ 4329.535124] LustreError: 134886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790784006 with bad export cookie 573325413957127499 [ 4329.557224] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4329.866828] Lustre: server umount lustre-MDT0001 complete [ 4339.482733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4340.267273] LustreError: 139081:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4340.297769] LustreError: 139081:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4340.380901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4345.674154] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4345.836494] LustreError: 139082:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4350.972340] LustreError: 139081:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4356.101746] LustreError: 139081:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4356.629229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4356.952512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4362.731445] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4371.443671] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4376.724255] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4383.688994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4384.219157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4392.392622] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4404.035554] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4410.397346] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4410.728985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4416.703265] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4417.006544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4427.107032] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4427.286027] Lustre: lustre-OST0001: new disk, initializing [ 4427.295943] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4427.305217] Lustre: 142467:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4427.415486] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4435.257670] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4436.110865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4436.130275] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4436.324076] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4444.711898] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4444.905977] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4449.673614] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4449.976487] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4455.944981] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93b15847d800.ost_server_uuid 50 [ 4457.824641] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff93b15847d800.ost_server_uuid in IDLE state after 0 sec [ 4467.183902] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4467.188491] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4467.199236] LustreError: Skipped 1 previous similar message [ 4467.200718] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4467.241909] Lustre: Skipped 1 previous similar message [ 4473.035427] Lustre: server umount lustre-OST0001 complete [ 4477.416487] LustreError: 140906:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4477.429521] LustreError: 140906:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4478.444644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4478.459511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4485.214105] Lustre: server umount lustre-OST0000 complete [ 4492.769907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4492.797825] Lustre: Skipped 1 previous similar message [ 4492.808735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4492.820775] Lustre: Skipped 8 previous similar messages [ 4497.441217] Lustre: server umount lustre-MDT0000 complete [ 4497.895242] LustreError: 139081:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4497.912322] LustreError: 139081:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4501.976264] LustreError: 139062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790784179 with bad export cookie 573325413957128381 [ 4501.985087] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4501.986266] LustreError: 139062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4502.485513] Lustre: server umount lustre-MDT0001 complete [ 4523.792955] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 4527.090170] Key type lgssc unregistered [ 4527.417574] LNet: 145470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4527.427788] LNetError: 145470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4527.447431] LNet: Removed LNI 192.168.201.126@tcp [ 4528.299604] Key type .llcrypt unregistered [ 4528.306268] Key type ._llcrypt unregistered [ 4581.179404] Key type ._llcrypt registered [ 4581.186264] Key type .llcrypt registered [ 4581.318586] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 4597.243536] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 4598.595229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4598.619110] alg: No test for adler32 (adler32-zlib) [ 4599.666118] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 4600.040108] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 4601.784504] Key type lgssc registered [ 4603.878666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4655.653461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4655.716802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4657.071565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4657.094211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4657.170546] Lustre: lustre-MDT0000: new disk, initializing [ 4657.262920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4657.286775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4662.371681] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4674.938905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4675.082815] Lustre: 149645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4675.175225] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4675.183413] Lustre: Skipped 1 previous similar message [ 4675.393759] Lustre: lustre-MDT0001: new disk, initializing [ 4675.600677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4675.694122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4675.723444] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4681.745932] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4692.566879] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4699.426753] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4708.208762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4708.526823] Lustre: lustre-OST0000: new disk, initializing [ 4708.539088] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4708.557880] Lustre: 151615:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4708.659613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4715.632916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4715.654055] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4715.808515] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4716.478220] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4727.247764] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4732.061136] Lustre: Failing over lustre-OST0000 [ 4732.378175] Lustre: server umount lustre-OST0000 complete [ 4749.793107] Lustre: 147859:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790784410/real 1790784410] req@0000000085ccecc5 x1877773414075520/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1790784426 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4749.870963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4751.968109] Lustre: 147856:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790784413/real 1790784413] req@000000002116756c x1877773414075776/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790784429 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4751.998572] Lustre: 147856:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4752.006686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4752.886462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4753.368878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4753.388365] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4757.451283] Lustre: 147859:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790784418/real 1790784418] req@00000000ea3dbaab x1877773414077440/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790784434 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4759.537680] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4762.593312] Lustre: 147856:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790784423/real 1790784423] req@00000000a816d99d x1877773414078720/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790784439 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4762.650726] Lustre: 147856:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4762.730748] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4762.731814] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4763.398243] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4768.266409] Lustre: Failing over lustre-MDT0000 [ 4768.585608] Lustre: server umount lustre-MDT0000 complete [ 4769.795989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4769.837938] Lustre: Skipped 1 previous similar message [ 4774.885600] LustreError: 149651:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4774.910638] LustreError: 149651:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4775.837512] LustreError: 149652:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4780.002902] LustreError: 150663:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4780.026382] LustreError: 150663:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4785.133550] LustreError: 149657:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4785.168841] LustreError: 149657:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4785.952171] Lustre: 147857:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790784446/real 1790784446] req@000000004685f46a x1877773414092800/t0(0) o400->MGC192.168.201.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790784462 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4785.980312] Lustre: 147857:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4785.993277] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4789.855664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4791.072440] LustreError: 152998:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4791.111592] LustreError: 152998:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4796.939310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4797.015410] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4797.257343] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4801.649340] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4802.023594] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4802.033661] Lustre: Skipped 1 previous similar message [ 4802.050755] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4802.100346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4812.265372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4812.270224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4812.285777] Lustre: Skipped 2 previous similar messages [ 4812.302801] Lustre: Skipped 1 previous similar message [ 4817.096573] Lustre: server umount lustre-OST0000 complete [ 4822.501254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4822.510631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4822.523267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4822.528266] Lustre: Skipped 1 previous similar message [ 4822.547108] Lustre: Skipped 1 previous similar message [ 4827.618634] LustreError: 149651:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4827.632042] LustreError: 149651:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 4829.526390] Lustre: server umount lustre-MDT0000 complete [ 4833.590351] LustreError: 151614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790784510 with bad export cookie 10355884540158787223 [ 4833.592987] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4833.599831] LustreError: 151614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4833.969983] Lustre: server umount lustre-MDT0001 complete [ 4847.513954] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 4850.775524] Key type lgssc unregistered [ 4851.112669] LNet: 155270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4851.129091] LNetError: 155270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4851.146032] LNet: Removed LNI 192.168.201.126@tcp [ 4852.079517] Key type .llcrypt unregistered [ 4852.081707] Key type ._llcrypt unregistered [ 4895.008813] Key type ._llcrypt registered [ 4895.020677] Key type .llcrypt registered [ 4895.219378] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 4896.674988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4896.955716] alg: No test for adler32 (adler32-zlib) [ 4898.230899] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 4898.580909] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 4900.397249] Key type lgssc registered [ 4901.849748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4911.014131] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4911.039083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4912.757959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4917.811528] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4922.341259] LustreError: 156809:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4922.407149] LustreError: 156809:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4927.472447] LustreError: 156808:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4930.100972] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4932.578760] LustreError: 156809:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4934.871270] LustreError: 156838:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4934.886253] LustreError: 156838:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4934.893670] LustreError: 156838:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 4935.212740] Lustre: server umount lustre-MDT0000 complete [ 4953.247929] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 4964.891467] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 5009.998561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5010.585292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5010.627871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5010.812937] Lustre: lustre-MDT0000: new disk, initializing [ 5010.981390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5011.035785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5017.942593] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5032.295888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5032.447693] Lustre: 160161:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5032.490680] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5032.493824] Lustre: Skipped 1 previous similar message [ 5032.546930] Lustre: lustre-MDT0001: new disk, initializing [ 5032.623365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5032.656556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5032.662829] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5039.024372] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5050.450597] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5057.400753] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5066.933232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5067.308358] Lustre: lustre-OST0000: new disk, initializing [ 5067.313620] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5067.323683] Lustre: 162129:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5067.440099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5074.236626] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5075.597303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5075.610760] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5075.753422] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5085.023543] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5090.833918] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5092.558372] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 12:12:48 (1790784768) [ 5365.736428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5365.749982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5365.780758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5367.779356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5367.797043] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5367.803844] Lustre: Skipped 1 previous similar message [ 5371.665694] Lustre: server umount lustre-MDT0000 complete [ 5372.903727] LustreError: 162136:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5372.934711] LustreError: 162136:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5375.740959] LustreError: 162140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790785052 with bad export cookie 16870521180651370076 [ 5375.743696] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5375.753643] LustreError: 162140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5378.019747] LustreError: 160167:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5378.020693] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5378.097075] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5378.101295] Lustre: Skipped 2 previous similar messages [ 5385.185031] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5385.198876] Lustre: Skipped 1 previous similar message [ 5390.305369] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5390.306921] LustreError: 162552:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5390.306979] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5390.364522] LustreError: 162552:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5390.686241] Lustre: server umount lustre-MDT0001 complete [ 5401.677315] Lustre: server umount lustre-OST0000 complete [ 5411.943186] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 5421.628826] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 5464.624413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5464.916693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5464.949493] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5465.087533] Lustre: lustre-MDT0000: new disk, initializing [ 5465.216637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5465.246718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5470.972646] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5482.192554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5482.335777] Lustre: 166800:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5482.434137] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5482.444976] Lustre: Skipped 1 previous similar message [ 5482.559288] Lustre: lustre-MDT0001: new disk, initializing [ 5482.647459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5482.703065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5482.712661] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5487.213994] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5497.700865] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5504.666759] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5511.990083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5512.315193] Lustre: lustre-OST0000: new disk, initializing [ 5512.319538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5512.327280] Lustre: 168773:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5512.410118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5519.123356] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5521.509971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5521.517468] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5521.627945] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5529.382674] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5534.557382] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5534.892244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5539.780727] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5540.058315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5542.370372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5542.370373] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5542.371072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5542.380811] Lustre: Skipped 1 previous similar message [ 5542.417993] Lustre: Skipped 1 previous similar message [ 5548.472778] Lustre: server umount lustre-OST0000 complete [ 5554.657220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5554.666649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5559.881482] Lustre: server umount lustre-MDT0000 complete [ 5562.849786] LustreError: 168767:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5562.874875] LustreError: 168767:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5564.616710] LustreError: 166794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790785241 with bad export cookie 16870521180651528906 [ 5564.618388] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5564.630127] LustreError: 166794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5564.984067] Lustre: server umount lustre-MDT0001 complete [ 5579.426919] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:20:54 (1790785254) [ 5597.422168] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 5609.367776] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 5648.321528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5648.591151] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5648.630328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5648.714598] Lustre: lustre-MDT0000: new disk, initializing [ 5648.791981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5648.807870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5653.228802] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5665.389363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5665.581902] Lustre: 173532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5665.936939] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5670.898977] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5681.010501] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5687.144332] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5694.713828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5695.056877] Lustre: lustre-OST0000: new disk, initializing [ 5695.059906] Lustre: Skipped 1 previous similar message [ 5695.064803] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5695.068812] Lustre: Skipped 2 previous similar messages [ 5695.074899] Lustre: 175512:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5701.547259] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5702.256973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5702.263578] Lustre: Skipped 1 previous similar message [ 5702.270308] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5702.374176] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5712.637290] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5723.108639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5723.118216] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5723.128178] Lustre: Skipped 1 previous similar message [ 5723.139501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5723.161701] Lustre: Skipped 6 previous similar messages [ 5727.120891] Lustre: server umount lustre-OST0000 complete [ 5732.323393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5737.664295] Lustre: server umount lustre-MDT0000 complete [ 5738.469980] LustreError: 173544:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5738.497526] LustreError: 173544:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5742.181247] LustreError: 173524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790785419 with bad export cookie 16870521180651529788 [ 5742.187735] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5742.190783] LustreError: 173524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5742.619864] Lustre: server umount lustre-MDT0001 complete [ 5751.977367] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:23:48 (1790785428) [ 5769.995931] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 5782.964671] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 5825.021705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5825.527524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5825.581711] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5825.740473] Lustre: lustre-MDT0000: new disk, initializing [ 5825.921955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5825.926477] Lustre: Skipped 2 previous similar messages [ 5825.969439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5830.846175] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5842.300847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5842.446369] Lustre: 179834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5842.757561] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5848.229496] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5857.858153] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5865.111816] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5872.218795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5872.465386] Lustre: 181812:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5878.964637] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5879.068507] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5880.136414] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5892.000209] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5904.872560] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5904.887717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5904.903489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5904.915027] Lustre: Skipped 4 previous similar messages [ 5904.935861] Lustre: Skipped 4 previous similar messages [ 5907.315465] Lustre: server umount lustre-OST0000 complete [ 5925.347976] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5925.879756] Lustre: server umount lustre-MDT0000 complete [ 5930.465385] LustreError: 179846:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5930.496724] LustreError: 179846:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5930.717141] LustreError: 179828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790785607 with bad export cookie 16870521180651530873 [ 5930.720152] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5930.732145] LustreError: 179828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5931.306252] Lustre: server umount lustre-MDT0001 complete [ 5950.734222] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 5953.863308] Key type lgssc unregistered [ 5954.169587] LNet: 183701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5954.173829] LNetError: 183701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5954.196239] LNet: Removed LNI 192.168.201.126@tcp [ 5955.063323] Key type .llcrypt unregistered [ 5955.065447] Key type ._llcrypt unregistered [ 5986.188315] Key type ._llcrypt registered [ 5986.190665] Key type .llcrypt registered [ 5986.259752] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 5987.604387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5987.802827] alg: No test for adler32 (adler32-zlib) [ 5989.077774] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 5989.374795] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 5991.040222] Key type lgssc registered [ 5993.275607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6005.779653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6005.856622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6007.831577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6014.235264] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6016.996160] LustreError: 185135:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6017.022187] LustreError: 185135:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6022.129895] LustreError: 185134:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6022.920955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6023.245466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6027.946576] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6037.295953] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6044.017384] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6052.900734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6053.648135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6060.696707] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6071.191906] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6077.238048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6097.903565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6097.918817] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6097.955088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6097.965184] Lustre: Skipped 1 previous similar message [ 6102.168450] Lustre: server umount lustre-OST0000 complete [ 6108.137914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6108.140797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6108.153302] Lustre: Skipped 2 previous similar messages [ 6108.176710] Lustre: Skipped 1 previous similar message [ 6113.250648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6113.261596] Lustre: Skipped 1 previous similar message [ 6113.613524] Lustre: server umount lustre-MDT0000 complete [ 6117.820820] LustreError: 185116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790785794 with bad export cookie 2679037840870807482 [ 6117.823094] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6117.829716] LustreError: 185116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6118.113617] Lustre: server umount lustre-MDT0001 complete [ 6131.832698] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 6135.184654] Key type lgssc unregistered [ 6135.462729] LNet: 188692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6135.474022] LNetError: 188692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6135.493296] LNet: Removed LNI 192.168.201.126@tcp [ 6136.465544] Key type .llcrypt unregistered [ 6136.467623] Key type ._llcrypt unregistered [ 6167.903537] Key type ._llcrypt registered [ 6167.906168] Key type .llcrypt registered [ 6168.011282] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 6169.444598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6169.576482] alg: No test for adler32 (adler32-zlib) [ 6171.054654] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 6171.326250] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 6173.116544] Key type lgssc registered [ 6174.641672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6184.735941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6184.766718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6186.583911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6192.954257] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6196.708142] LustreError: 190126:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6196.716778] LustreError: 190126:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6201.831794] LustreError: 190145:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6202.357694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6202.589478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6208.106485] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6218.024902] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6225.759907] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6234.236237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6234.453527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6240.772829] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6250.017240] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6256.007659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6262.743758] Lustre: lustre-OST0000: Not available for connect from 192.168.201.26@tcp (stopping) [ 6263.276873] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6263.287212] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6263.304209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6266.338560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6266.353121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6266.355693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6268.561864] Lustre: server umount lustre-OST0000 complete [ 6277.438194] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff93b16054b800.ost_server_uuid 50 [ 6279.799264] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6295.011103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6295.029083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6295.073345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6295.084534] Lustre: Skipped 2 previous similar messages [ 6297.060508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6300.148781] Lustre: server umount lustre-MDT0000 complete [ 6302.184882] LustreError: 190145:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6302.212687] LustreError: 190145:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6304.204148] LustreError: 190106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790785981 with bad export cookie 12482090440571065743 [ 6304.230737] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6304.516626] Lustre: server umount lustre-MDT0001 complete [ 6315.947466] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 12:33:11 (1790785991) [ 6323.933583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6324.283874] LustreError: 194064:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6324.350389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6328.547192] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6329.825661] LustreError: 194065:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6338.591851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6338.927501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6344.769860] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6354.979688] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6361.488126] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6369.553734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6369.700320] LustreError: 195888:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6369.716805] LustreError: 195888:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6369.935233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6375.410790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6375.538321] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6385.881724] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6395.728042] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6395.978384] Lustre: lustre-OST0001: new disk, initializing [ 6395.983122] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6395.990291] Lustre: 196909:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6396.093231] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6404.728169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6404.756438] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6404.938630] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6406.001454] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6420.492338] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6424.557089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6424.570964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6424.595211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6424.610855] Lustre: Skipped 2 previous similar messages [ 6430.225964] Lustre: server umount lustre-OST0000 complete [ 6430.702746] LustreError: 195889:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6430.761984] LustreError: 195889:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6438.864978] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6439.140346] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6443.110705] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6443.307755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6461.409198] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6461.429038] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6461.435255] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6461.440136] Lustre: Skipped 2 previous similar messages [ 6461.449588] Lustre: Skipped 4 previous similar messages [ 6465.822608] Lustre: server umount lustre-OST0001 complete [ 6476.954742] Lustre: server umount lustre-MDT0000 complete [ 6480.894119] LustreError: 194045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790786157 with bad export cookie 12482090440571066814 [ 6480.899594] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6480.903031] LustreError: 194045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6481.296952] Lustre: server umount lustre-MDT0001 complete [ 6509.137591] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 12:36:24 (1790786184) [ 6517.708961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6518.040237] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6518.094315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6518.393189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6524.376587] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6534.439623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6534.643603] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6534.703565] Lustre: 200349:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6539.941703] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6549.037638] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6555.535315] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6563.820264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6564.011639] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6572.359408] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6572.700168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6580.288705] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6588.366048] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6588.462583] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6588.646033] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6588.657282] Lustre: Skipped 2 previous similar messages [ 6595.994621] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6596.793945] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6604.399928] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6607.335829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6607.346338] Lustre: Skipped 2 previous similar messages [ 6607.351755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6607.359682] Lustre: Skipped 5 previous similar messages [ 6613.639724] Lustre: server umount lustre-OST0000 complete [ 6615.467341] LustreError: 202591:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6615.491901] LustreError: 202591:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [ 6632.930654] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6632.942392] LustreError: Skipped 2 previous similar messages [ 6636.842945] Lustre: server umount lustre-OST0001 complete [ 6642.668261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6642.684597] Lustre: Skipped 3 previous similar messages [ 6642.695354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6642.703723] Lustre: Skipped 5 previous similar messages [ 6647.578654] Lustre: server umount lustre-MDT0000 complete [ 6652.584119] LustreError: 202217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790786329 with bad export cookie 12482090440571068221 [ 6652.598186] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6652.599350] LustreError: 202217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6653.240952] Lustre: server umount lustre-MDT0001 complete [ 6681.963107] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 12:39:17 (1790786357) [ 6700.305481] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 6711.270627] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 6753.850067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6754.177908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6754.230704] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6754.346556] Lustre: lustre-MDT0000: new disk, initializing [ 6754.474193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6754.515973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6759.424478] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6771.283173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6771.456453] Lustre: 208446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6771.490893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6771.493536] Lustre: Skipped 1 previous similar message [ 6771.562911] Lustre: lustre-MDT0001: new disk, initializing [ 6771.682020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6771.702624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6777.542523] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6789.572735] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6798.470155] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6807.090388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6807.445737] Lustre: lustre-OST0000: new disk, initializing [ 6807.448775] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6807.454255] Lustre: 210419:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6814.372949] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6815.283995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6815.297431] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6815.487550] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6825.089284] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6829.039457] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6829.213392] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6833.650494] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6833.889138] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6836.198550] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6836.199909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6836.202362] LustreError: Skipped 1 previous similar message [ 6836.212264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6836.239959] Lustre: Skipped 2 previous similar messages [ 6836.261856] Lustre: Skipped 3 previous similar messages [ 6842.380475] Lustre: server umount lustre-OST0000 complete [ 6852.747229] Lustre: server umount lustre-MDT0000 complete [ 6856.675681] LustreError: 208458:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6856.696827] LustreError: 208458:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 6857.116079] LustreError: 208438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790786534 with bad export cookie 12482090440571069999 [ 6857.119597] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6857.129423] LustreError: 208438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6857.615630] Lustre: server umount lustre-MDT0001 complete [ 6867.961599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6874.279877] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6884.057364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6889.305635] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6897.776454] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6904.157954] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6911.511592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6918.536106] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6928.165947] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6934.595217] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6934.881561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6941.574547] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6941.960343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6944.772201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6944.780841] LustreError: Skipped 1 previous similar message [ 6950.273951] Lustre: server umount lustre-OST0000 complete [ 6958.121872] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6961.868358] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6992.252336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6999.948993] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7000.877796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7000.893764] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7000.899269] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7008.048125] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7010.115404] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7016.421387] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7016.427741] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7016.436499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7016.436508] Lustre: Skipped 10 previous similar messages [ 7016.449513] Lustre: Skipped 5 previous similar messages [ 7016.480695] LustreError: Skipped 1 previous similar message [ 7021.389749] Lustre: server umount lustre-OST0000 complete [ 7032.087750] Lustre: server umount lustre-MDT0000 complete [ 7036.301129] LustreError: 216747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790786713 with bad export cookie 12482090440571070881 [ 7036.309893] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7036.590532] Lustre: server umount lustre-MDT0001 complete [ 7045.968266] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 12:45:21 (1790786721) [ 7054.073679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7054.641535] LustreError: 218605:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7054.674314] LustreError: 218605:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7054.753853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7054.762180] Lustre: Skipped 6 previous similar messages [ 7059.971683] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7070.368384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7076.002194] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7084.844910] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7091.999915] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7099.908105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7107.368822] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7115.512916] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7120.389818] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7125.985206] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 7130.437129] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7130.623967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7138.390180] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7138.649414] Lustre: lustre-OST0001: new disk, initializing [ 7138.654362] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7138.659401] Lustre: 222012:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7144.596651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7144.604664] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7144.835817] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7146.326724] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7157.153563] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7163.171441] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7163.438429] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7167.958475] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7168.195636] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7170.541694] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7184.352142] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7184.458598] Lustre: server umount lustre-OST0001 complete [ 7191.605239] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7191.782234] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7196.215659] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7196.428927] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7226.027674] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7231.497325] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7231.520841] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7231.522389] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7231.551495] Lustre: Skipped 2 previous similar messages [ 7232.465994] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7240.338506] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7242.289090] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7259.106351] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7259.359354] Lustre: server umount lustre-OST0001 complete [ 7278.562108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7278.571806] Lustre: Skipped 6 previous similar messages [ 7278.577112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7278.583145] Lustre: Skipped 17 previous similar messages [ 7284.649982] Lustre: server umount lustre-OST0000 complete [ 7295.441159] Lustre: server umount lustre-MDT0000 complete [ 7299.502083] LustreError: 218586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790786976 with bad export cookie 12482090440571071854 [ 7299.504037] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7299.518674] LustreError: 218586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7306.050657] Lustre: server umount lustre-MDT0001 complete [ 7331.356228] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 12:50:07 (1790787007) [ 7337.006578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7337.354187] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7337.414495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7341.728413] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7350.237431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7350.337746] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7350.361509] Lustre: 227260:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7354.917870] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7361.482590] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7367.387228] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7373.657900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7373.850041] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7380.066817] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7387.568157] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7396.347314] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7396.446467] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7402.741233] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7409.813576] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7411.717503] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7417.259246] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7417.628243] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7423.687470] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7424.016525] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7431.854478] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93b15814c000.ost_server_uuid 50 [ 7434.098070] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff93b15814c000.ost_server_uuid in IDLE state after 0 sec [ 7435.644692] Lustre: Permanently deactivating lustre-OST0001 [ 7435.659933] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7441.492970] Lustre: Permanently reactivating lustre-OST0001 [ 7441.513759] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7441.519288] Lustre: Skipped 2 previous similar messages [ 7450.081597] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7450.095303] LustreError: Skipped 6 previous similar messages [ 7452.096335] Lustre: server umount lustre-OST0001 complete [ 7455.214941] LustreError: 229103:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7455.229794] LustreError: 229103:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [ 7461.884683] Lustre: server umount lustre-OST0000 complete [ 7472.593100] Lustre: server umount lustre-MDT0000 complete [ 7476.122298] LustreError: 227252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790787153 with bad export cookie 12482090440571073142 [ 7476.127847] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7476.132984] LustreError: 227252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7476.560628] Lustre: server umount lustre-MDT0001 complete [ 7501.405814] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 12:52:57 (1790787177) [ 7510.442896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7510.992055] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7511.049944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7511.071567] Lustre: Skipped 2 previous similar messages [ 7517.309729] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7526.553235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7526.743885] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7526.793615] Lustre: 233954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7532.585664] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7540.539656] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7546.494244] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7553.441133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7553.632212] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7553.655800] Lustre: Permanently deactivating lustre-OST0000 [ 7553.670346] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7559.251265] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7561.245469] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7566.574135] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7573.481416] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7573.578855] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7573.801449] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7573.808678] Lustre: Skipped 11 previous similar messages [ 7579.800655] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7580.891689] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7587.953250] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7590.100236] Lustre: Permanently reactivating lustre-OST0000 [ 7590.121336] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7590.125814] Lustre: Skipped 2 previous similar messages [ 7597.111028] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7597.113643] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 36 seconds. I think it's dead, and I am evicting it. exp 00000000834711eb, cur 1790787274 deadline 1790787268 last 1790787238 [ 7597.121125] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7597.159462] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7597.224284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7608.696252] Lustre: server umount lustre-OST0001 complete [ 7619.630909] Lustre: server umount lustre-OST0000 complete [ 7631.360754] Lustre: server umount lustre-MDT0000 complete [ 7635.024672] LustreError: 233946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790787312 with bad export cookie 12482090440571074815 [ 7635.028168] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7635.029758] LustreError: 233946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7635.197059] Lustre: server umount lustre-MDT0001 complete [ 7645.226653] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 7647.967292] Key type lgssc unregistered [ 7648.186750] LNet: 238832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7648.192277] LNetError: 238832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7648.210898] LNet: Removed LNI 192.168.201.126@tcp [ 7649.232223] Key type .llcrypt unregistered [ 7649.235836] Key type ._llcrypt unregistered [ 7674.116394] Key type ._llcrypt registered [ 7674.118042] Key type .llcrypt registered [ 7674.236992] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 7675.619070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7676.010795] alg: No test for adler32 (adler32-zlib) [ 7676.984253] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 7677.240719] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 7678.936223] Key type lgssc registered [ 7680.749375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7691.938434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7691.957750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7693.700105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7699.784424] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7703.010598] LustreError: 240315:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7703.022631] LustreError: 240315:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7708.136393] LustreError: 240316:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7709.145060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7709.308264] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7709.389574] Lustre: 240304:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7709.404396] Lustre: Permanently deactivating lustre-MDT0001 [ 7709.437339] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7709.744383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7711.040647] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7711.050850] LustreError: 240346:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7711.065856] LustreError: 240346:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7711.078424] LustreError: 240346:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 1, failed: rc = -108 [ 7714.416142] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7724.541113] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7732.046722] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7738.298360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7738.404257] LustreError: 242157:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7738.498734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7739.588981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7744.860159] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7745.009762] LustreError: 242159:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7745.038428] LustreError: 242159:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7750.115274] LustreError: 242158:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7751.942107] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7755.256482] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7757.736861] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7757.886368] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7762.968720] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7762.971850] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7763.564151] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7770.369110] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7773.500566] Lustre: Permanently reactivating lustre-MDT0001 [ 7773.523924] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7773.527233] Lustre: Skipped 2 previous similar messages [ 7779.775803] Lustre: lustre-MDT0001: Client 0e696ca9-b5db-4299-ad6f-1063fca8633c (at 192.168.201.26@tcp) reconnecting [ 7785.802171] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7787.483735] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7792.771457] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7792.944804] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7797.208218] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7797.386762] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7799.894258] Lustre: Permanently deactivating lustre-MDT0001 [ 7799.909616] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7799.916934] Lustre: Skipped 2 previous similar messages [ 7807.619579] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7817.192241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7817.195602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7817.222774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7819.238229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7819.242446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7819.247690] Lustre: Skipped 2 previous similar messages [ 7819.259149] Lustre: Skipped 2 previous similar messages [ 7821.096311] Lustre: server umount lustre-MDT0000 complete [ 7824.357132] LustreError: 240315:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7824.389908] LustreError: 240315:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7825.581255] LustreError: 240295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790787502 with bad export cookie 10197196816769760302 [ 7825.586356] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7825.589148] LustreError: 240295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7826.178410] Lustre: server umount lustre-MDT0001 complete [ 7833.145549] Lustre: server umount lustre-OST0000 complete [ 7846.260157] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:58:42 (1790787522) [ 7857.198284] Lustre: server umount lustre-OST0001 complete [ 7863.232400] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 7872.647955] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 7909.633932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7909.919443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7909.924943] Lustre: Skipped 2 previous similar messages [ 7909.949518] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7910.038928] Lustre: lustre-MDT0000: new disk, initializing [ 7910.133774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7910.146892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7915.146766] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7925.242611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7925.355563] Lustre: 248573:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7925.426369] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7925.434212] Lustre: Skipped 1 previous similar message [ 7925.527365] Lustre: lustre-MDT0001: new disk, initializing [ 7925.653945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7925.696742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7925.706162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7929.650396] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7939.776241] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7945.855699] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7953.050717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7953.301484] Lustre: lustre-OST0000: new disk, initializing [ 7953.305456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7953.312724] Lustre: 250543:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7953.402196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7960.423753] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7962.748174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7962.756751] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7962.918703] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7972.088403] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7975.417897] LustreError: 248578:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7985.483322] LustreError: 248578:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7985.537726] LustreError: 251318:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7986.264490] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7986.522469] Lustre: lustre-OST0001: new disk, initializing [ 7986.534528] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7986.543683] Lustre: 251668:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7986.663802] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7992.684854] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7992.945150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7992.954320] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7993.083979] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7995.561081] LustreError: 251318:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8003.268107] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8004.703334] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8008.673567] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8008.677894] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8008.687072] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8008.687080] Lustre: Skipped 1 previous similar message [ 8008.706987] Lustre: Skipped 1 previous similar message [ 8013.457410] Lustre: server umount lustre-OST0001 complete [ 8013.805194] LustreError: 250528:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8013.815249] LustreError: 250528:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8034.281140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8034.293159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8034.311870] Lustre: Skipped 1 previous similar message [ 8034.321821] Lustre: Skipped 1 previous similar message [ 8036.640461] Lustre: server umount lustre-OST0000 complete [ 8043.495994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8043.504343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8043.513369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8047.704556] Lustre: server umount lustre-MDT0000 complete [ 8049.635359] LustreError: 249699:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8049.651407] LustreError: 249699:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 8051.943543] LustreError: 248564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790787728 with bad export cookie 10197196816769763109 [ 8051.957056] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8051.963474] LustreError: 248564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8058.433162] Lustre: server umount lustre-MDT0001 complete [ 8069.870660] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 8072.219729] Key type lgssc unregistered [ 8072.490559] LNet: 253699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8072.508041] LNetError: 253699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8073.576498] LNet: Removed LNI 192.168.201.126@tcp [ 8074.560142] Key type .llcrypt unregistered [ 8074.563857] Key type ._llcrypt unregistered [ 8115.338645] Key type ._llcrypt registered [ 8115.340811] Key type .llcrypt registered [ 8115.451067] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 8116.730646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8116.798423] alg: No test for adler32 (adler32-zlib) [ 8117.941139] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 8118.245339] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 8119.920554] Key type lgssc registered [ 8121.228386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8130.804329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8130.850712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8132.182139] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8132.210590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8132.367480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8136.558623] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8145.582248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8145.762683] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8145.812117] Lustre: 255922:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8146.309538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8150.552723] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8157.266908] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8163.030488] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8170.118166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8170.242708] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8170.425165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8175.693227] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8179.877331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8182.211906] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8236.516383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8236.517063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8236.536872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8236.556394] Lustre: Skipped 1 previous similar message [ 8236.570610] Lustre: Skipped 1 previous similar message [ 8239.900382] Lustre: server umount lustre-OST0000 complete [ 8244.209370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8255.548705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8255.946158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8257.020217] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8257.036995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8257.050892] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8257.061824] Lustre: Skipped 1 previous similar message [ 8262.765244] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8271.544351] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8318.434911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8318.444163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8318.457720] Lustre: Skipped 1 previous similar message [ 8318.464855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8318.481948] Lustre: Skipped 1 previous similar message [ 8320.406780] Lustre: server umount lustre-OST0000 complete [ 8325.604844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8325.614378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8325.648940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8328.676153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8328.679654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8328.703868] Lustre: Skipped 1 previous similar message [ 8331.468659] Lustre: server umount lustre-MDT0000 complete [ 8335.770392] LustreError: 255915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790788012 with bad export cookie 1403463560495476734 [ 8335.785273] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8335.796461] LustreError: 255915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8336.470750] Lustre: server umount lustre-MDT0001 complete [ 8347.788943] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 8350.445478] Key type lgssc unregistered [ 8350.722859] LNet: 261360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8350.728366] LNetError: 261360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8350.746698] LNet: Removed LNI 192.168.201.126@tcp [ 8351.608281] Key type .llcrypt unregistered [ 8351.611515] Key type ._llcrypt unregistered [ 8375.747452] Key type ._llcrypt registered [ 8375.750182] Key type .llcrypt registered [ 8375.872699] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 8377.391382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8377.441109] alg: No test for adler32 (adler32-zlib) [ 8378.608974] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 8378.954814] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 8380.641986] Key type lgssc registered [ 8381.832222] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8389.700837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8389.730534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8391.359922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8395.613447] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8396.779934] LustreError: 262810:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8396.810827] LustreError: 262810:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8401.891143] LustreError: 262809:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8404.323609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8404.806839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8410.240758] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8422.605285] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8430.897390] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8438.750902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8439.165340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8446.353817] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8455.464307] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8461.722909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8490.465390] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8490.468254] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8490.491760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8492.520111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8492.525066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8492.548153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8492.572738] Lustre: Skipped 1 previous similar message [ 8496.168226] Lustre: server umount lustre-OST0000 complete [ 8502.245225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8502.258568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8502.283282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8506.283819] Lustre: server umount lustre-MDT0000 complete [ 8507.877102] LustreError: 262809:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8507.885292] LustreError: 262809:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8509.866951] LustreError: 262791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790788186 with bad export cookie 15610636798069457166 [ 8509.879576] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8509.884746] LustreError: 262791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8510.256477] Lustre: server umount lustre-MDT0001 complete [ 8520.963991] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 8523.755516] Key type lgssc unregistered [ 8524.080512] LNet: 267134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8524.086821] LNetError: 267134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8524.108904] LNet: Removed LNI 192.168.201.126@tcp [ 8525.003934] Key type .llcrypt unregistered [ 8525.012903] Key type ._llcrypt unregistered [ 8543.951363] Key type ._llcrypt registered [ 8543.954440] Key type .llcrypt registered [ 8544.063820] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 8545.214309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8545.337990] alg: No test for adler32 (adler32-zlib) [ 8546.331583] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 8546.528849] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 8548.168869] Key type lgssc registered [ 8549.845655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8558.064210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8558.080903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8559.659554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8564.312389] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8569.827955] LustreError: 268374:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8569.863403] LustreError: 268374:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8574.557580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8575.104539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8580.618618] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8588.714717] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8594.647514] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8601.968324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8602.390425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8604.489047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8608.564626] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8615.139723] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8630.243719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8630.246235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8630.257088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8630.273616] Lustre: Skipped 1 previous similar message [ 8634.933630] Lustre: server umount lustre-OST0000 complete [ 8640.500617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8640.513950] Lustre: Skipped 1 previous similar message [ 8640.521472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8640.532937] Lustre: Skipped 1 previous similar message [ 8645.609332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8645.621689] Lustre: Skipped 1 previous similar message [ 8650.726740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8653.280219] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8653.528799] Lustre: server umount lustre-MDT0000 complete [ 8655.850728] LustreError: 268375:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8658.407898] LustreError: 268356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790788335 with bad export cookie 11092727904993255511 [ 8658.415846] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8658.420890] LustreError: 268356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8658.777407] Lustre: server umount lustre-MDT0001 complete [ 8670.420724] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 8673.358897] Key type lgssc unregistered [ 8673.577670] LNet: 272054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8673.581031] LNetError: 272054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8673.601825] LNet: Removed LNI 192.168.201.126@tcp [ 8674.649970] Key type .llcrypt unregistered [ 8674.653889] Key type ._llcrypt unregistered [ 8699.544576] Key type ._llcrypt registered [ 8699.545862] Key type .llcrypt registered [ 8699.652700] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 8700.939312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8701.031096] alg: No test for adler32 (adler32-zlib) [ 8702.074538] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 8702.358248] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 8704.056474] Key type lgssc registered [ 8705.371373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8713.364120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8713.378405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8714.806081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8718.789911] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8724.962910] LustreError: 273486:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8724.978421] LustreError: 273486:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8728.212162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8728.616118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8733.295155] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8742.268614] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8748.507557] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8757.055498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8757.522609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8760.582239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8763.770788] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8772.374557] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8796.645571] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8796.651882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8796.673668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8799.715770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8799.721930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8799.747822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8799.751809] Lustre: Skipped 1 previous similar message [ 8802.476469] Lustre: server umount lustre-OST0000 complete [ 8809.955452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8809.963636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8809.990026] Lustre: Skipped 1 previous similar message [ 8810.003391] Lustre: Skipped 1 previous similar message [ 8812.622959] Lustre: server umount lustre-MDT0000 complete [ 8815.075817] LustreError: 274489:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8815.105255] LustreError: 274489:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8817.477162] LustreError: 273466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790788494 with bad export cookie 4748882693876610021 [ 8817.492891] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8817.495893] LustreError: 273466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8817.898076] Lustre: server umount lustre-MDT0001 complete [ 8831.948064] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 8835.192563] Key type lgssc unregistered [ 8835.465344] LNet: 277796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8835.478645] LNetError: 277796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8835.498776] LNet: Removed LNI 192.168.201.126@tcp [ 8836.526270] Key type .llcrypt unregistered [ 8836.528901] Key type ._llcrypt unregistered [ 8855.101053] Key type ._llcrypt registered [ 8855.104657] Key type .llcrypt registered [ 8855.202176] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 8856.287350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8856.310784] alg: No test for adler32 (adler32-zlib) [ 8857.321433] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 8857.516425] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 8859.240210] Key type lgssc registered [ 8860.430449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8867.116863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8867.132475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8868.758189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8873.393155] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8878.055952] LustreError: 279048:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8878.093892] LustreError: 279048:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8881.857334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8882.307717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8887.006488] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8895.640835] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8902.039977] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8909.171560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8909.648107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8914.747875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8915.851083] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8923.911772] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8940.513924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8940.520707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8940.532195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8942.564906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8942.565530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8942.578365] Lustre: Skipped 1 previous similar message [ 8943.708503] Lustre: server umount lustre-OST0000 complete [ 8947.686682] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8947.698723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8947.708254] Lustre: Skipped 1 previous similar message [ 8947.729237] Lustre: Skipped 1 previous similar message [ 8952.813170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8952.818305] Lustre: Skipped 1 previous similar message [ 8953.564213] Lustre: server umount lustre-MDT0000 complete [ 8956.958961] LustreError: 279015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790788634 with bad export cookie 7858817502565219423 [ 8956.968604] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8957.378360] Lustre: server umount lustre-MDT0001 complete [ 8967.452352] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 8969.774912] Key type lgssc unregistered [ 8970.001578] LNet: 282750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8970.017604] LNetError: 282750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8970.038877] LNet: Removed LNI 192.168.201.126@tcp [ 8970.730732] Key type .llcrypt unregistered [ 8970.732510] Key type ._llcrypt unregistered [ 8982.145768] Key type ._llcrypt registered [ 8982.147903] Key type .llcrypt registered [ 8982.256953] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9000.012807] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 9014.585727] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 9016.024366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9016.120028] alg: No test for adler32 (adler32-zlib) [ 9017.464318] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 9017.984517] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 9019.768968] Key type lgssc registered [ 9021.400521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9059.278828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9059.340900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9060.642955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9060.668914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9060.755341] Lustre: lustre-MDT0000: new disk, initializing [ 9060.874263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9060.891448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9065.519273] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 9076.162539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9076.301748] Lustre: 286312:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9076.347298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9076.353985] Lustre: Skipped 1 previous similar message [ 9076.409364] Lustre: lustre-MDT0001: new disk, initializing [ 9076.506618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9076.565167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9076.580688] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9081.454959] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 9091.104939] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9097.467117] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9104.974509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9105.340445] Lustre: lustre-OST0000: new disk, initializing [ 9105.344290] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9105.349473] Lustre: 288281:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9105.458679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9111.532906] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 9114.734379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9114.740492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9114.859938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9120.379715] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9124.269056] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9124.474820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9128.883546] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9129.080161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9135.591167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9135.599864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9135.602349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9135.608286] Lustre: Skipped 1 previous similar message [ 9140.710500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9140.742391] Lustre: Skipped 1 previous similar message [ 9145.312166] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9145.677471] Lustre: server umount lustre-OST0000 complete [ 9150.946134] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9150.969067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9150.980996] Lustre: Skipped 2 previous similar messages [ 9156.070361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9156.076026] Lustre: Skipped 1 previous similar message [ 9157.063196] Lustre: server umount lustre-MDT0000 complete [ 9161.143744] LustreError: 286305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790788838 with bad export cookie 10728681586495539410 [ 9161.148473] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9161.152661] LustreError: 286305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9161.565509] Lustre: server umount lustre-MDT0001 complete [ 9170.954441] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 13:20:46 (1790788846) [ 9177.538830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9178.159768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9182.633784] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 9183.204116] LustreError: 290694:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9183.221879] LustreError: 290694:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9188.327841] LustreError: 290693:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9191.316529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9191.687754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9196.228638] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 9204.153758] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9210.180365] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9217.340991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9217.687872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9224.531498] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 9233.409933] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9244.654949] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9244.662971] LustreError: Skipped 1 previous similar message [ 9244.667402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9244.684230] Lustre: Skipped 1 previous similar message [ 9244.690404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9246.689728] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9246.697230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9250.524568] Lustre: server umount lustre-OST0000 complete [ 9256.930830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9256.934934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9256.947763] Lustre: Skipped 1 previous similar message [ 9256.958814] Lustre: Skipped 3 previous similar messages [ 9268.192253] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9268.348845] Lustre: server umount lustre-MDT0000 complete [ 9271.277213] LustreError: 290688:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9271.306591] LustreError: 290688:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9272.790286] LustreError: 290674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790788949 with bad export cookie 10728681586495540299 [ 9272.796111] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9272.804500] LustreError: 290674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9273.169790] Lustre: server umount lustre-MDT0001 complete [ 9286.666135] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 9289.847445] Key type lgssc unregistered [ 9290.083173] LNet: 294230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9290.088297] LNetError: 294230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9290.101929] LNet: Removed LNI 192.168.201.126@tcp [ 9290.749396] Key type .llcrypt unregistered [ 9290.751621] Key type ._llcrypt unregistered [ 9319.129840] Key type ._llcrypt registered [ 9319.132150] Key type .llcrypt registered [ 9319.231268] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 9336.557765] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 9337.564340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9337.668605] alg: No test for adler32 (adler32-zlib) [ 9338.662784] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 9338.859315] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 9340.512329] Key type lgssc registered [ 9341.716899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9395.290817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9395.339341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9396.733890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9396.786687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9396.933706] Lustre: lustre-MDT0000: new disk, initializing [ 9397.025174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9397.041372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9402.172250] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 9413.502321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9413.687539] Lustre: 298071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9413.760654] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9413.763896] Lustre: Skipped 1 previous similar message [ 9413.843732] Lustre: lustre-MDT0001: new disk, initializing [ 9413.943878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9413.982417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9413.996893] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9418.945979] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 9430.791302] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9437.649468] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9446.806715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9447.342361] Lustre: lustre-OST2710: new disk, initializing [ 9447.347537] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9447.359509] Lustre: 300038:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9447.501017] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9454.803634] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 9457.205535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9457.215913] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9457.352625] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9464.035348] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9471.884311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9472.092467] Lustre: lustre-OST03e8: new disk, initializing [ 9472.096790] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9472.102301] Lustre: 301107:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9472.210387] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9478.452222] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 9481.873053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9481.879580] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9481.964470] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9487.467231] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9496.050472] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9496.283347] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9500.832861] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9501.058638] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9505.216861] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9505.397392] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9510.231221] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9510.400404] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9516.513550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9516.517558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9516.525082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9518.066585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9518.079576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9518.083070] Lustre: Skipped 1 previous similar message [ 9518.088185] Lustre: Skipped 2 previous similar messages [ 9520.327692] Lustre: server umount lustre-MDT0000 complete [ 9524.046469] LustreError: 298065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790789201 with bad export cookie 8150589343328799578 [ 9524.055687] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9524.059525] LustreError: 298065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9524.321483] Lustre: server umount lustre-MDT0001 complete [ 9538.150799] Lustre: server umount lustre-OST2710 complete [ 9551.329618] Lustre: server umount lustre-OST03e8 complete [ 9571.510325] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 9581.204696] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 9621.874615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9622.163446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9622.250829] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9622.364824] Lustre: lustre-MDT0000: new disk, initializing [ 9622.483173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9622.509810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9627.790261] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 9639.135077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9639.264349] Lustre: 306362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9639.329709] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9639.334282] Lustre: Skipped 1 previous similar message [ 9639.433063] Lustre: lustre-MDT0001: new disk, initializing [ 9639.562352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9639.600889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9639.617811] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9645.116752] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 9657.452136] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9665.138968] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9673.413043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9673.711967] Lustre: lustre-OST0000: new disk, initializing [ 9673.714934] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9673.718809] Lustre: 308330:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9673.806356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9681.357912] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 9681.537418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9681.552709] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9681.748039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9691.897316] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9697.425178] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9697.624490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9703.180493] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9703.407408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9707.495819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9707.500253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9707.527833] Lustre: Skipped 1 previous similar message [ 9707.530253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9707.558257] Lustre: Skipped 2 previous similar messages [ 9711.789465] Lustre: server umount lustre-OST0000 complete [ 9717.730841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9717.763707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9717.768663] Lustre: Skipped 2 previous similar messages [ 9717.799122] Lustre: Skipped 1 previous similar message [ 9722.756735] Lustre: server umount lustre-MDT0000 complete [ 9722.866761] LustreError: 306373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9722.882394] LustreError: 306373:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 9727.014800] LustreError: 306369:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9727.036629] LustreError: 306369:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9727.039902] LustreError: 308329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790789404 with bad export cookie 8150589343328801258 [ 9727.057289] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9727.108751] LustreError: 308329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9727.535204] Lustre: server umount lustre-MDT0001 complete [ 9739.158790] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 13:30:13 (1790789413) [ 9741.106244] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9743.107868] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 13:30:18 (1790789418) [ 9749.718578] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 9779.673248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9779.997554] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9780.010443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9780.266273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9786.364867] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 9796.734839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9796.876177] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9796.918786] Lustre: 312190:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9802.698394] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 9825.077278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9825.397839] Lustre: Denying initial registration attempt from nid 192.168.201.126@tcp, specified as failover [ 9825.403960] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9825.414150] LustreError: 313647:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9825.422428] LustreError: 313647:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9825.426724] LustreError: 313647:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9825.508858] Lustre: server umount lustre-OST0000 complete [ 9825.513230] LustreError: 313647:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9832.929833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9832.937794] LustreError: Skipped 1 previous similar message [ 9832.942865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9832.956442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9838.415931] Lustre: server umount lustre-MDT0000 complete [ 9842.148888] LustreError: 312203:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9842.181419] LustreError: 312203:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9842.565064] LustreError: 312182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790789519 with bad export cookie 8150589343328802140 [ 9842.571224] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9842.573292] LustreError: 312182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9843.224969] Lustre: server umount lustre-MDT0001 complete [ 9858.870591] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 9863.788560] Key type lgssc unregistered [ 9864.128747] LNet: 314767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9864.138207] LNetError: 314767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9864.155726] LNet: Removed LNI 192.168.201.126@tcp [ 9865.097387] Key type .llcrypt unregistered [ 9865.099096] Key type ._llcrypt unregistered [ 9879.553485] Key type ._llcrypt registered [ 9879.555396] Key type .llcrypt registered [ 9879.686231] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 9880.859300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9880.949331] alg: No test for adler32 (adler32-zlib) [ 9882.120574] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [ 9882.444394] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 9884.112170] Key type lgssc registered [ 9885.450735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9936.209866] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 9947.150107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9947.178331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9948.492691] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9948.512729] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9948.664649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9953.557815] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 9962.977266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9963.045628] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9963.070980] Lustre: 317548:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9963.349132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9968.743278] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 9976.311329] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9981.557558] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10001.346941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10001.777161] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10001.988428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10010.261439] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [10018.239772] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10021.348470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10021.360784] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10021.401454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10021.862471] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10021.874370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10026.560793] Lustre: server umount lustre-OST0000 complete [10032.100428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10032.104328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10032.127715] Lustre: Skipped 1 previous similar message [10032.155500] Lustre: Skipped 2 previous similar messages [10037.661176] Lustre: server umount lustre-MDT0000 complete [10042.026773] LustreError: 317542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790789719 with bad export cookie 12638255513217240606 [10042.044885] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10042.060892] LustreError: 317542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10042.594141] Lustre: server umount lustre-MDT0001 complete [10054.825245] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [10058.602438] Key type lgssc unregistered [10058.993485] LNet: 321122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10058.997609] LNetError: 321122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10059.012986] LNet: Removed LNI 192.168.201.126@tcp [10060.094617] Key type .llcrypt unregistered [10060.096845] Key type ._llcrypt unregistered [10088.715247] Key type ._llcrypt registered [10088.719639] Key type .llcrypt registered [10088.816639] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [10090.243744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10090.367886] alg: No test for adler32 (adler32-zlib) [10091.380957] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [10091.741954] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [10093.432419] Key type lgssc registered [10095.032519] Lustre: Echo OBD driver; http://www.lustre.org/ [10102.862532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10102.896840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10104.514519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10109.169364] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [10114.033772] LustreError: 322557:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10114.070496] LustreError: 322557:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10118.702867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10119.127827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10123.871467] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [10131.182488] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10136.629402] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10147.053127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10147.811487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10156.186650] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [10166.241459] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10179.507210] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.26@tcp (stopping) [10179.565797] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10179.584599] Lustre: Skipped 1 previous similar message [10182.279981] Lustre: server umount lustre-MDT0000 complete [10184.615752] LustreError: 324468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10186.558619] LustreError: 322538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790789863 with bad export cookie 13301362137156903699 [10186.566064] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10189.723810] LustreError: 322553:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10189.745888] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.26@tcp (stopping) [10189.752977] LustreError: 322553:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10189.774258] Lustre: Skipped 3 previous similar messages [10189.798950] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10189.824124] Lustre: Skipped 1 previous similar message [10191.846851] LustreError: 322551:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10191.849788] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10191.863552] LustreError: 322551:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10191.881342] Lustre: Skipped 1 previous similar message [10193.048029] Lustre: server umount lustre-MDT0001 complete [10200.282285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10211.202480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10212.320369] LustreError: 326136:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [10212.321847] LustreError: 321854:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000026781270 x1877779171803520/t0(0) o250->MGC192.168.201.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10212.396665] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb897ee63de117ef7 [10212.436625] Lustre: MGC192.168.201.126@tcp: Connection restored to 0@lo (at 0@lo) [10212.874927] LustreError: 324388:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10212.950853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10213.022068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10219.929595] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [10222.195465] LustreError: 326149:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10222.200146] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10222.223335] LustreError: 326149:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10234.073756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10234.534953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10239.340337] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [10240.007930] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10248.381550] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10250.703867] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10258.659660] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10259.990690] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10269.153537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10269.165168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10269.178183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10270.177692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10274.467275] Lustre: server umount lustre-OST0000 complete [10280.418086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10280.439357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10280.465987] Lustre: Skipped 1 previous similar message [10280.477681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10280.480496] Lustre: Skipped 2 previous similar messages [10285.974301] Lustre: server umount lustre-MDT0000 complete [10286.057378] LustreError: 326155:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10286.101486] LustreError: 326155:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [10290.791210] LustreError: 322536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790789967 with bad export cookie 13301362137156910839 [10290.792804] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10290.809690] LustreError: 322536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10291.411761] Lustre: server umount lustre-MDT0001 complete [10308.295600] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [10312.033886] Key type lgssc unregistered [10312.354496] LNet: 328845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10312.367344] LNetError: 328845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10312.393528] LNet: Removed LNI 192.168.201.126@tcp [10313.641592] Key type .llcrypt unregistered [10313.650339] Key type ._llcrypt unregistered [10345.020705] Key type ._llcrypt registered [10345.025405] Key type .llcrypt registered [10345.272714] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [10346.204749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10346.334951] alg: No test for adler32 (adler32-zlib) [10347.936707] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [10348.393778] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [10350.240261] Key type lgssc registered [10352.356834] Lustre: Echo OBD driver; http://www.lustre.org/ [10364.491856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10364.503912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10366.085369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10371.681224] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [10376.679873] LustreError: 330276:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10376.726269] LustreError: 330276:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10381.050528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10381.468952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10386.127887] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [10393.783445] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10399.026727] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10406.373831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10406.381074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10406.388687] Lustre: Skipped 1 previous similar message [10406.399671] Lustre: Skipped 1 previous similar message [10407.400657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10407.406747] Lustre: Skipped 1 previous similar message [10409.078163] Lustre: server umount lustre-MDT0000 complete [10412.513781] LustreError: 330273:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10412.536334] LustreError: 330273:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10413.866222] LustreError: 330257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790790090 with bad export cookie 17056767169021175346 [10413.882077] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10414.249908] Lustre: server umount lustre-MDT0001 complete [10430.612588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10431.028473] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10431.100492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10431.432444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10436.139264] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [10445.419168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10445.530711] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10445.554880] Lustre: 332864:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10445.870816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10450.527431] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [10458.631929] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10465.773861] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10475.952673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10476.405480] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10476.425878] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10476.440343] LustreError: 334749:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [10476.449243] LustreError: 334749:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10476.452109] LustreError: 334749:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10476.593636] Lustre: server umount lustre-OST0000 complete [10476.595589] LustreError: 334749:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10489.082335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10489.462656] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10489.807503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10496.691883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10497.698125] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [10507.149795] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10516.126337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10516.293737] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10516.452905] Lustre: lustre-OST0001: new disk, initializing [10516.460403] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10516.470742] Lustre: 336295:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10516.642956] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10522.298199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10522.318275] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10522.439936] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10524.449890] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [10533.203329] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10537.961372] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10537.972680] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10537.975835] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10537.989133] Lustre: Skipped 1 previous similar message [10538.017524] Lustre: Skipped 1 previous similar message [10542.781961] Lustre: server umount lustre-OST0001 complete [10543.081201] LustreError: 335531:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10543.100437] LustreError: 335531:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10548.208798] LustreError: 335531:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10548.212557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10548.232262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10548.232278] LustreError: Skipped 1 previous similar message [10548.244511] LustreError: 335531:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10548.285727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10548.289055] Lustre: Skipped 1 previous similar message [10548.340022] Lustre: Skipped 1 previous similar message [10553.327431] LustreError: 335215:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10553.369183] LustreError: 335215:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10553.661844] Lustre: server umount lustre-OST0000 complete [10558.437422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10558.439386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10558.454816] LustreError: Skipped 1 previous similar message [10558.461715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10558.495785] Lustre: Skipped 1 previous similar message [10564.105300] Lustre: server umount lustre-MDT0000 complete [10568.471671] LustreError: 333603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790790245 with bad export cookie 17056767169021175899 [10568.474917] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10568.482428] LustreError: 333603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10568.995174] Lustre: server umount lustre-MDT0001 complete [10580.572818] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [10583.951756] Key type lgssc unregistered [10584.324433] LNet: 338255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10584.332771] LNetError: 338255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10585.381909] LNet: Removed LNI 192.168.201.126@tcp [10586.429961] Key type .llcrypt unregistered [10586.435658] Key type ._llcrypt unregistered [10670.066542] Key type ._llcrypt registered [10670.070945] Key type .llcrypt registered [10670.210814] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [10686.729465] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [10687.823576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10687.903261] alg: No test for adler32 (adler32-zlib) [10689.024856] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [10689.271944] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [10690.968215] Key type lgssc registered [10692.167402] Lustre: Echo OBD driver; http://www.lustre.org/ [10739.034045] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10739.082958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10740.483699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10740.515522] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10740.672848] Lustre: lustre-MDT0000: new disk, initializing [10740.807780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10740.846407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10748.041338] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [10762.076912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10762.316876] Lustre: 343204:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10762.460354] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10762.466323] Lustre: Skipped 1 previous similar message [10762.598907] Lustre: lustre-MDT0001: new disk, initializing [10762.817438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10762.870048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10762.873471] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10769.244590] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [10781.543145] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10788.454033] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10795.705677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10796.164103] Lustre: lustre-OST0000: new disk, initializing [10796.170759] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10796.179452] Lustre: 345175:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10796.314469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10801.740223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10801.751315] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10801.836792] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10802.433176] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [10812.040917] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10816.877313] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10817.059206] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10821.427384] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10821.631709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10827.755427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10827.757691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10827.771287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10827.778142] LustreError: Skipped 1 previous similar message [10827.795297] Lustre: Skipped 1 previous similar message [10829.975569] Lustre: server umount lustre-OST0000 complete [10834.401381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10834.408872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10834.430759] Lustre: Skipped 1 previous similar message [10834.439033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10837.986292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10837.988710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10840.242922] Lustre: server umount lustre-MDT0000 complete [10844.285118] LustreError: 343197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790790521 with bad export cookie 6575110571951892231 [10844.293514] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10844.299440] LustreError: 343197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10844.599558] Lustre: server umount lustre-MDT0001 complete [10854.718985] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 13:48:50 (1790790530) [10864.898468] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 13:49:00 (1790790540) [10876.269434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10877.100718] LustreError: 347935:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10877.132030] LustreError: 347935:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10877.201081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10882.532340] LustreError: 347936:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10882.800381] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [10887.660606] LustreError: 347935:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10892.770877] LustreError: 347936:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10892.963716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10893.447338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10897.411619] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [10905.084597] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10912.045546] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10918.684073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10919.061239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10924.767908] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [10932.651584] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10944.512482] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10946.447219] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10948.300697] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10950.617914] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10953.660422] Lustre: Failing over lustre-MDT0000 [10954.088646] Lustre: server umount lustre-MDT0000 complete [10954.215955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10954.223683] LustreError: 348989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10954.225866] Lustre: Skipped 2 previous similar messages [10954.253913] LustreError: 348989:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10964.453366] LustreError: 347936:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10964.478313] LustreError: 347936:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [10970.465125] Lustre: 341421:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790790631/real 1790790631] req@00000000b1b8d159 x1877779798892928/t0(0) o400->MGC192.168.201.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790790647 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10970.509772] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10974.950459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10980.834669] LustreError: 341417:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a48e80da x1877779798898432/t0(0) o250->MGC192.168.201.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10980.880916] LustreError: 348663:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10980.905340] LustreError: 348663:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [10981.097625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10981.138918] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10981.802400] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10986.513214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10986.557529] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10986.597956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10986.676233] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [10997.988879] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10999.750215] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11004.136922] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11006.286110] Lustre: Failing over lustre-MDT0000 [11006.690837] Lustre: server umount lustre-MDT0000 complete [11006.949154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11006.953644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11017.201169] LustreError: 347936:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11017.227923] LustreError: 347936:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [11028.448317] Lustre: 341421:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790790689/real 1790790689] req@00000000aaa514c7 x1877779798916480/t0(0) o400->MGC192.168.201.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790790705 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11028.470867] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11029.038417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11039.031905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11039.090603] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11041.218174] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11043.937711] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [11044.348386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11044.369383] Lustre: Skipped 2 previous similar messages [11044.427419] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [11044.480408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11054.285730] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11056.027095] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11064.811186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11064.812279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11064.815019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11064.815029] Lustre: Skipped 1 previous similar message [11064.843047] Lustre: Skipped 4 previous similar messages [11069.134579] Lustre: server umount lustre-MDT0000 complete [11072.761478] LustreError: 347915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790790749 with bad export cookie 6575110571951894947 [11072.771591] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11073.075637] Lustre: server umount lustre-MDT0001 complete [11087.737783] Lustre: server umount lustre-OST0000 complete [11104.485035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11105.106256] LustreError: 354295:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11105.121384] LustreError: 354295:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 23 previous similar messages [11105.197079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11110.146769] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [11118.360357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11123.124047] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [11131.118532] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11137.868149] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11146.834157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11150.531090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11154.389932] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [11162.749302] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11174.373693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11174.378273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11174.391966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11174.422846] Lustre: Skipped 3 previous similar messages [11180.253231] Lustre: server umount lustre-OST0000 complete [11184.621347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11184.641516] Lustre: Skipped 3 previous similar messages [11190.191201] Lustre: server umount lustre-MDT0000 complete [11195.172791] LustreError: 354320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790790872 with bad export cookie 6575110571951895682 [11195.180441] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11195.188117] LustreError: 354320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11195.817842] Lustre: server umount lustre-MDT0001 complete [11209.908522] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [11212.804628] Key type lgssc unregistered [11213.212972] LNet: 357831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11213.221864] LNetError: 357831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11213.244859] LNet: Removed LNI 192.168.201.126@tcp [11214.171796] Key type .llcrypt unregistered [11214.174766] Key type ._llcrypt unregistered [11245.099698] Key type ._llcrypt registered [11245.101687] Key type .llcrypt registered [11245.213855] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [11246.518464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11246.661871] alg: No test for adler32 (adler32-zlib) [11248.030181] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [11248.331178] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [11250.016302] Key type lgssc registered [11251.658898] Lustre: Echo OBD driver; http://www.lustre.org/ [11260.661990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11260.721745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11262.421786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11267.209192] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [11272.685655] LustreError: 359365:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11272.711149] LustreError: 359365:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11276.673758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11277.360882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11283.807752] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [11292.908350] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11300.546601] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11309.987727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11310.557950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11318.668661] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [11327.995339] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11332.107423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11347.425463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11347.463485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11347.467854] Lustre: Skipped 2 previous similar messages [11347.478265] Lustre: Skipped 2 previous similar messages [11351.020645] Lustre: server umount lustre-MDT0000 complete [11352.560043] LustreError: 359365:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11355.590860] LustreError: 359347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790791032 with bad export cookie 14806234693842695522 [11355.602655] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11355.609781] LustreError: 359347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11356.121165] Lustre: server umount lustre-MDT0001 complete [11370.181277] Lustre: server umount lustre-OST0000 complete [11384.869052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11385.558126] LustreError: 363100:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11385.578921] LustreError: 363100:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11385.669068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11390.249898] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [11390.949035] LustreError: 363101:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11395.050507] LustreError: 363100:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11398.990932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11399.382351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11404.812876] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [11414.552920] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11423.020554] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11433.040352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11433.627994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11441.281307] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [11452.080726] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11455.178266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11471.336710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11471.348497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11471.368709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11475.937157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11475.944441] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11477.642110] Lustre: server umount lustre-MDT0000 complete [11481.068244] LustreError: 363096:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11481.097705] LustreError: 363096:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11483.206589] LustreError: 363081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790791160 with bad export cookie 14806234693842696642 [11483.209270] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11483.227144] LustreError: 363081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11483.852088] Lustre: server umount lustre-MDT0001 complete [11498.313221] Lustre: server umount lustre-OST0000 complete [11518.528634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11519.114417] LustreError: 367016:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11519.205742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11524.359576] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [11536.758026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11543.782543] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [11552.821689] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11560.982718] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11569.535539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11569.918551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11569.923540] Lustre: Skipped 1 previous similar message [11576.674331] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [11586.116245] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11589.416136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11597.796159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11597.812460] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11597.854278] Lustre: Skipped 1 previous similar message [11597.885656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11597.897317] Lustre: Skipped 2 previous similar messages [11599.854528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11602.721369] Lustre: server umount lustre-OST0000 complete [11609.057408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11609.070911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11609.091020] Lustre: Skipped 1 previous similar message [11609.102370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11609.116464] Lustre: Skipped 2 previous similar messages [11620.332401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11620.348988] Lustre: Skipped 5 previous similar messages [11622.369663] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11622.656477] Lustre: server umount lustre-MDT0000 complete [11625.443800] LustreError: 367017:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11625.476073] LustreError: 367017:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11629.362541] LustreError: 366998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790791306 with bad export cookie 14806234693842697664 [11629.382957] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11630.576620] Lustre: server umount lustre-MDT0001 complete [11644.372452] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [11647.882383] Key type lgssc unregistered [11648.144168] LNet: 370563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11648.151536] LNetError: 370563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11648.168936] LNet: Removed LNI 192.168.201.126@tcp [11649.153353] Key type .llcrypt unregistered [11649.160145] Key type ._llcrypt unregistered [11682.837237] Key type ._llcrypt registered [11682.843830] Key type .llcrypt registered [11683.080821] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [11684.469429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11684.502383] alg: No test for adler32 (adler32-zlib) [11685.850834] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [11686.177890] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [11687.960845] Key type lgssc registered [11688.995443] Lustre: Echo OBD driver; http://www.lustre.org/ [11697.512773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11697.561436] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11697.578386] LustreError: 372038:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11697.607757] LustreError: 372038:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11697.615642] LustreError: 372038:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11697.621972] LustreError: 372038:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11697.635800] LustreError: 372038:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11705.874584] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11705.883406] LustreError: 372435:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11705.916888] LustreError: 372435:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11705.925846] LustreError: 372435:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11705.938585] LustreError: 372435:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11705.960193] LustreError: 372435:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11728.383185] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [11731.685858] Key type lgssc unregistered [11732.073649] LNet: 373439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11732.081526] LNetError: 373439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11732.124331] LNet: Removed LNI 192.168.201.126@tcp [11732.997366] Key type .llcrypt unregistered [11732.999123] Key type ._llcrypt unregistered [11754.755469] Key type ._llcrypt registered [11754.758699] Key type .llcrypt registered [11754.837089] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [11771.466803] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [11773.167669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11773.446526] alg: No test for adler32 (adler32-zlib) [11774.800312] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [11775.316194] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [11777.080157] Key type lgssc registered [11778.986398] Lustre: Echo OBD driver; http://www.lustre.org/ [11824.706270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11824.770028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11826.102992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11826.182755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11826.353513] Lustre: lustre-MDT0000: new disk, initializing [11826.522353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11826.542778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11831.761576] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [11844.150952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11844.346544] Lustre: 376613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11844.433822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11844.438181] Lustre: Skipped 1 previous similar message [11844.591873] Lustre: lustre-MDT0001: new disk, initializing [11844.771659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11844.840385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11844.858215] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11850.643487] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [11862.029518] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11868.717915] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11876.088764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11876.349373] Lustre: lustre-OST0000: new disk, initializing [11876.355636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11876.362836] Lustre: 378581:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11876.443655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11884.004845] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [11886.193905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11886.200492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11886.408454] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11897.234307] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11902.898027] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11903.133958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11911.330845] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11911.646201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11917.283322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11917.305165] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11917.336346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11921.070604] Lustre: server umount lustre-OST0000 complete [11926.504997] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11926.514120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11926.536299] Lustre: Skipped 1 previous similar message [11926.545938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11926.549678] Lustre: Skipped 1 previous similar message [11927.524747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11927.559556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11927.574851] Lustre: Skipped 1 previous similar message [11932.800544] Lustre: server umount lustre-MDT0000 complete [11937.775563] LustreError: 376618:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11937.811875] LustreError: 376618:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11937.846592] LustreError: 376604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790791614 with bad export cookie 16955871010201222025 [11937.857364] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11937.859640] LustreError: 376604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11938.418497] Lustre: server umount lustre-MDT0001 complete [11947.674952] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11949.495690] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 14:07:05 (1790791625) [11957.189594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11957.553094] LustreError: 381042:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11957.674028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11962.698518] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [11962.855525] LustreError: 381043:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11967.973859] LustreError: 381042:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11972.607842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11973.111567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11979.580788] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [11991.741348] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11998.443621] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12005.663085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12006.102611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12014.001307] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [12023.502504] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12034.272822] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12034.484426] Lustre: lustre-OST0001: new disk, initializing [12034.490367] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12034.499088] Lustre: 383883:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12034.654286] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12041.397354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12041.407348] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12041.556411] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12042.890396] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [12054.238526] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12062.186955] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12062.191120] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12062.202252] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12062.202823] Lustre: Skipped 1 previous similar message [12065.993657] Lustre: server umount lustre-OST0001 complete [12067.314565] LustreError: 382866:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12067.339903] LustreError: 382866:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12077.545046] LustreError: 382864:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12077.575566] LustreError: 382864:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12085.217799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12085.224114] LustreError: Skipped 1 previous similar message [12085.229927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12085.233893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12085.245208] Lustre: Skipped 1 previous similar message [12085.261376] Lustre: Skipped 3 previous similar messages [12089.354577] Lustre: server umount lustre-OST0000 complete [12095.969684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12095.979495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12095.991698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12095.999419] Lustre: Skipped 2 previous similar messages [12099.397437] Lustre: server umount lustre-MDT0000 complete [12103.137376] LustreError: 381042:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12103.164176] LustreError: 381042:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [12103.464984] LustreError: 383882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790791780 with bad export cookie 16955871010201222907 [12103.477281] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12103.481448] LustreError: 383882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12109.962593] Lustre: server umount lustre-MDT0001 complete [12123.569223] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [12126.883908] Key type lgssc unregistered [12127.202725] LNet: 385870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12127.208531] LNetError: 385870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12127.232976] LNet: Removed LNI 192.168.201.126@tcp [12128.123900] Key type .llcrypt unregistered [12128.125756] Key type ._llcrypt unregistered [12195.728551] Key type ._llcrypt registered [12195.732222] Key type .llcrypt registered [12196.125573] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [12199.177396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12199.221902] alg: No test for adler32 (adler32-zlib) [12200.503935] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [12201.906080] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [12203.762882] Key type lgssc registered [12206.307078] Lustre: Echo OBD driver; http://www.lustre.org/ [12218.312141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12218.345605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12219.669510] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12219.701727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12219.859756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12226.266686] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [12238.163753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12238.282831] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12238.321372] Lustre: 388688:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12238.694069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12243.703982] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [12251.894715] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12257.343805] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12259.302466] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12259.307057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12259.334206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12261.861349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12261.863891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12265.518322] Lustre: server umount lustre-MDT0000 complete [12270.078226] LustreError: 388679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790791947 with bad export cookie 15651441015451894102 [12270.089050] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12270.100952] LustreError: 388679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12270.573701] Lustre: server umount lustre-MDT0001 complete [12281.177597] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 14:12:37 (1790791957) [12289.008659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12289.456097] LustreError: 391199:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12289.478149] LustreError: 391199:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12289.520913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12294.655131] LustreError: 391200:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12294.682072] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [12299.755200] LustreError: 391199:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12304.739231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12304.871696] LustreError: 391200:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12305.213236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12310.632307] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [12320.197576] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12327.218933] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12334.730990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12334.934886] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12335.146340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12342.176576] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [12351.248229] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12360.650040] Lustre: Permanently deactivating lustre-OST0000 [12360.685828] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12368.429947] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12371.021936] LustreError: 394190:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12371.035992] LustreError: 394190:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12371.046452] LustreError: 394190:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12375.755663] LustreError: 394239:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12375.763773] LustreError: 394239:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12380.961312] Lustre: server umount lustre-OST0000 complete [12384.524789] LustreError: 394488:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12384.542667] LustreError: 394488:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12387.308829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12387.322500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12387.345473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12387.352253] Lustre: Skipped 1 previous similar message [12390.369854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12390.372416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12390.399744] Lustre: Skipped 1 previous similar message [12395.491087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12395.500184] Lustre: Skipped 1 previous similar message [12397.704730] Lustre: server umount lustre-MDT0000 complete [12400.609429] LustreError: 391196:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12400.618516] LustreError: 391196:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12401.909469] LustreError: 391179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790792078 with bad export cookie 15651441015451894676 [12401.915274] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12401.919644] LustreError: 391179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12411.304198] Lustre: server umount lustre-MDT0001 complete [12420.689382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12426.544485] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [12429.588948] LustreError: 395646:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12436.443640] Lustre: 395985:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.126@tcp [12438.816634] Lustre: server umount MGS complete [12451.856267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12452.587935] LustreError: 396709:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12452.608784] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12452.622401] Lustre: Skipped 1 previous similar message [12452.690276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12457.478650] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [12467.098776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12467.414754] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12472.678110] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [12481.372564] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12488.238497] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12490.367939] Lustre: Permanently reactivating lustre-OST0000 [12490.384195] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12490.396323] Lustre: Skipped 2 previous similar messages [12517.344386] LustreError: 398443:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12525.559181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12526.168866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12526.178726] Lustre: Skipped 1 previous similar message [12534.702411] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [12544.761293] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12552.166562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12552.169562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12552.251163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12557.599774] Lustre: server umount lustre-OST0000 complete [12563.427566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12563.456809] Lustre: Skipped 1 previous similar message [12568.563802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12568.573571] Lustre: Skipped 4 previous similar messages [12569.246089] Lustre: server umount lustre-MDT0000 complete [12573.666182] LustreError: 396710:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12573.687798] LustreError: 396710:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12574.463542] LustreError: 396691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790792251 with bad export cookie 15651441015451895733 [12574.475230] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12574.485880] LustreError: 396691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12575.505693] Lustre: server umount lustre-MDT0001 complete [12591.540726] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [12595.111247] Key type lgssc unregistered [12595.697196] LNet: 400977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12595.710809] LNetError: 400977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12595.728086] LNet: Removed LNI 192.168.201.126@tcp [12597.129426] Key type .llcrypt unregistered [12597.131917] Key type ._llcrypt unregistered [12620.278273] Key type ._llcrypt registered [12620.280866] Key type .llcrypt registered [12620.381934] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [12636.846902] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [12638.359087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12638.438698] alg: No test for adler32 (adler32-zlib) [12639.681248] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [12640.063522] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [12641.785466] Key type lgssc registered [12643.538037] Lustre: Echo OBD driver; http://www.lustre.org/ [12693.836589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12693.923507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12695.438353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12695.507858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12695.658143] Lustre: lustre-MDT0000: new disk, initializing [12695.862050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12695.906850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12701.273865] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [12713.362692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12713.552415] Lustre: 404155:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12713.682897] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12713.686256] Lustre: Skipped 1 previous similar message [12713.848823] Lustre: lustre-MDT0001: new disk, initializing [12714.060368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12714.152295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12714.168487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12719.353849] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [12728.813368] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12735.677294] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12746.691042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12747.449074] Lustre: lustre-OST0000: new disk, initializing [12747.454604] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12747.463685] Lustre: 406124:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12747.691088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12756.411300] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [12757.633019] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12757.651543] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12757.844231] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12767.721976] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12774.393292] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12774.652616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12780.589865] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12780.690621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12783.585439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12783.589461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12783.592206] LustreError: Skipped 1 previous similar message [12783.604194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12783.608647] Lustre: Skipped 1 previous similar message [12783.631569] Lustre: Skipped 1 previous similar message [12788.744613] Lustre: server umount lustre-OST0000 complete [12793.827680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12793.831680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12793.873031] Lustre: Skipped 1 previous similar message [12798.951597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12798.961145] Lustre: Skipped 2 previous similar messages [12799.594776] Lustre: server umount lustre-MDT0000 complete [12803.849090] LustreError: 404148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790792480 with bad export cookie 10998029421102414823 [12803.869317] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12804.296350] Lustre: server umount lustre-MDT0001 complete [12814.212655] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12815.931776] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 14:21:31 (1790792491) [12822.738677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12823.252119] LustreError: 408585:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12823.272141] LustreError: 408585:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12823.322044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12828.033821] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [12828.643175] LustreError: 408586:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12833.765154] LustreError: 408585:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12836.651274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12836.990275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12842.293579] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [12850.325750] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12855.700229] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12862.721092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12863.171780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12869.870181] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [12878.743857] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12880.745459] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12883.649326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12892.135744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12892.147827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12892.148344] Lustre: Skipped 1 previous similar message [12892.160244] Lustre: Skipped 1 previous similar message [12895.413211] Lustre: server umount lustre-OST0000 complete [12902.369517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12902.375680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12902.380414] Lustre: Skipped 1 previous similar message [12902.386976] Lustre: Skipped 1 previous similar message [12905.585144] Lustre: server umount lustre-MDT0000 complete [12907.491382] LustreError: 408580:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12907.516716] LustreError: 408580:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12909.682160] LustreError: 408566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790792586 with bad export cookie 10998029421102415698 [12909.697334] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12909.701035] LustreError: 408566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12910.036587] Lustre: server umount lustre-MDT0001 complete [12923.458514] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [12926.892782] Key type lgssc unregistered [12927.223658] LNet: 412311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12927.227853] LNetError: 412311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12927.238811] LNet: Removed LNI 192.168.201.126@tcp [12928.214717] Key type .llcrypt unregistered [12928.217753] Key type ._llcrypt unregistered [12953.154747] Key type ._llcrypt registered [12953.158162] Key type .llcrypt registered [12953.294155] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [12953.647158] Key type .llcrypt unregistered [12953.652843] Key type ._llcrypt unregistered [12973.189946] Key type ._llcrypt registered [12973.193028] Key type .llcrypt registered [12973.342917] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [12974.594807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12974.661731] alg: No test for adler32 (adler32-zlib) [12975.656307] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [12975.894500] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [12977.584681] Key type lgssc registered [12978.757376] Lustre: Echo OBD driver; http://www.lustre.org/ [12986.412324] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12986.427241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12988.200616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12993.016322] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [12998.625723] LustreError: 414537:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12998.658111] LustreError: 414537:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13001.326471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13001.789633] LustreError: 414538:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13001.812258] LustreError: 414538:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13001.819680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13006.819569] LustreError: 414537:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13006.849171] LustreError: 414537:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13009.074872] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [13011.944281] LustreError: 414538:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13011.973093] LustreError: 414538:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13013.002958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13017.610924] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13018.085092] LustreError: 414537:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13018.119201] LustreError: 414537:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13025.757808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13026.137162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13031.408270] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [13041.647255] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13041.652736] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13041.670206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13046.760108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13046.768483] Lustre: Skipped 1 previous similar message [13047.579661] Lustre: server umount lustre-OST0000 complete [13051.875190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13051.880691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13051.890335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13051.890344] Lustre: Skipped 1 previous similar message [13051.899115] Lustre: Skipped 2 previous similar messages [13056.994084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13056.998647] Lustre: Skipped 2 previous similar messages [13057.722630] Lustre: server umount lustre-MDT0000 complete [13061.929742] LustreError: 414518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790792738 with bad export cookie 12198229111973649562 [13061.942801] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13061.944326] LustreError: 414518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13062.495133] Lustre: server umount lustre-MDT0001 complete [13075.015548] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [13078.176126] Key type lgssc unregistered [13078.513472] LNet: 417685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13078.533484] LNetError: 417685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13078.558337] LNet: Removed LNI 192.168.201.126@tcp [13079.528457] Key type .llcrypt unregistered [13079.533718] Key type ._llcrypt unregistered [13109.999799] Key type ._llcrypt registered [13110.006371] Key type .llcrypt registered [13110.131294] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [13111.348775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13111.437468] alg: No test for adler32 (adler32-zlib) [13112.579696] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [13112.883317] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [13114.584279] Key type lgssc registered [13116.039346] Lustre: Echo OBD driver; http://www.lustre.org/ [13124.552365] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13124.569045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13140.256521] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13145.376475] LustreError: 419090:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.126@tcp: failed processing log, type 4: rc = -110 [13172.064267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13179.923791] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [13187.035307] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13193.822105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13194.365136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13194.432975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13199.318731] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [13199.841346] LustreError: 419116:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13199.870877] LustreError: 419116:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13204.964609] LustreError: 419115:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13206.002142] LustreError: 420128:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13206.031615] LustreError: 420128:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13208.416245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13208.867097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13213.771439] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [13221.541623] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13227.830328] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13239.778105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13239.785764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13239.793420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13239.826376] Lustre: Skipped 1 previous similar message [13241.788865] Lustre: server umount lustre-OST0000 complete [13250.020533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13250.022593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13250.031849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13250.031860] Lustre: Skipped 1 previous similar message [13250.044026] Lustre: Skipped 1 previous similar message [13251.810835] Lustre: server umount lustre-MDT0000 complete [13255.139784] LustreError: 420134:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13255.162470] LustreError: 420134:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13255.899326] LustreError: 419096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790792932 with bad export cookie 7450550578669756014 [13255.903195] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13255.914435] LustreError: 419096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13256.185969] Lustre: server umount lustre-MDT0001 complete [13267.584394] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [13270.279332] Key type lgssc unregistered [13270.515656] LNet: 422657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13270.520127] LNetError: 422657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13270.559092] LNet: Removed LNI 192.168.201.126@tcp [13271.499659] Key type .llcrypt unregistered [13271.504809] Key type ._llcrypt unregistered [13296.976019] Key type ._llcrypt registered [13296.978053] Key type .llcrypt registered [13297.174390] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [13298.324444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13298.401832] alg: No test for adler32 (adler32-zlib) [13299.462104] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [13299.644148] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [13301.329564] Key type lgssc registered [13302.433429] Lustre: Echo OBD driver; http://www.lustre.org/ [13309.983057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13310.004341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13311.456536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13316.444080] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [13321.700233] LustreError: 424093:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13321.711387] LustreError: 424093:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13325.231780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13325.459398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13330.276433] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [13338.444570] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13345.252631] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13353.232466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13353.707812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13356.783679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13360.464571] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [13369.021591] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13375.461569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13375.492683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13375.501522] Lustre: Skipped 2 previous similar messages [13375.527239] Lustre: Skipped 2 previous similar messages [13376.939256] Lustre: server umount lustre-MDT0000 complete [13376.962665] LustreError: 424089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13379.537333] LustreError: 424074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.26@tcp arrived at 1790793056 with bad export cookie 13762382477461025981 [13379.546564] LustreError: 424074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13380.582169] LustreError: 424094:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13380.622724] LustreError: 424094:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13381.505821] LustreError: 424073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790793058 with bad export cookie 13762382477461025085 [13381.515619] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13385.698061] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13385.713300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13385.713658] LustreError: 425594:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13385.728630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13385.742246] LustreError: 425594:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13387.832864] Lustre: server umount lustre-OST0000 complete [13391.844280] LustreError: 425291:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13391.856869] LustreError: 425291:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13394.897850] Lustre: server umount lustre-MDT0001 complete [13407.840689] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [13410.767212] Key type lgssc unregistered [13411.162593] LNet: 427776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13411.167293] LNetError: 427776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13411.180667] LNet: Removed LNI 192.168.201.126@tcp [13412.058729] Key type .llcrypt unregistered [13412.060888] Key type ._llcrypt unregistered [13439.068259] Key type ._llcrypt registered [13439.072175] Key type .llcrypt registered [13439.194715] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [13440.086337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13440.227683] alg: No test for adler32 (adler32-zlib) [13441.466835] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [13441.745753] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [13443.440225] Key type lgssc registered [13445.011205] Lustre: Echo OBD driver; http://www.lustre.org/ [13452.765596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13452.796202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13454.476075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13459.024636] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [13464.043302] LustreError: 429207:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13464.076630] LustreError: 429207:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13468.086350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13468.372594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13473.381337] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [13481.238833] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13487.711576] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13496.473051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13496.906336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13499.952386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13503.638977] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [13512.237274] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13518.311962] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13518.326213] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13518.340403] Lustre: Skipped 1 previous similar message [13518.363264] Lustre: Skipped 1 previous similar message [13520.317225] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.26@tcp (stopping) [13523.428992] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13523.443795] Lustre: Skipped 1 previous similar message [13524.057824] Lustre: server umount lustre-MDT0001 complete [13525.418239] LustreError: 429202:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13528.546739] LustreError: 429203:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13528.585599] LustreError: 429203:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13530.596052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13530.603280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13530.618575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13533.679782] LustreError: 429207:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13536.804636] Lustre: server umount lustre-OST0000 complete [13538.789447] LustreError: 429208:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13541.173704] Lustre: server umount lustre-MDT0000 complete [13555.998608] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [13559.485783] Key type lgssc unregistered [13559.785842] LNet: 432889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13559.799362] LNetError: 432889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13559.816517] LNet: Removed LNI 192.168.201.126@tcp [13560.970631] Key type .llcrypt unregistered [13560.972724] Key type ._llcrypt unregistered [13587.919206] Key type ._llcrypt registered [13587.922163] Key type .llcrypt registered [13588.088539] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [13605.123186] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [13606.136552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13606.336186] alg: No test for adler32 (adler32-zlib) [13607.357419] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [13607.561113] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [13609.240383] Key type lgssc registered [13610.870755] Lustre: Echo OBD driver; http://www.lustre.org/ [13652.037366] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [13670.153381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13670.193280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13671.505838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13671.537589] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13671.641520] Lustre: lustre-MDT0000: new disk, initializing [13671.766615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13671.776119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13675.771625] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [13691.153647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13691.386924] Lustre: lustre-OST0000: new disk, initializing [13691.390381] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13691.393736] Lustre: Skipped 1 previous similar message [13691.397727] Lustre: 437943:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13691.451698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13697.717794] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [13701.216170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13701.233459] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13701.289873] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13707.532071] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13723.019646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13723.172250] Lustre: 436925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13723.253415] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13723.324451] Lustre: lustre-MDT0001: new disk, initializing [13723.438102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13723.493086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13723.503600] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13728.136527] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [13740.097762] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13740.379485] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13746.132651] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13746.490156] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13756.387735] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13756.391707] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13756.402360] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13756.414767] Lustre: Skipped 1 previous similar message [13757.369293] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.26@tcp (stopping) [13757.378325] Lustre: Skipped 1 previous similar message [13758.551305] Lustre: server umount lustre-MDT0001 complete [13766.629394] LustreError: 439269:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13766.668864] LustreError: 439269:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [13771.756250] LustreError: 436938:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13771.777996] LustreError: 436938:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13776.867398] LustreError: 436934:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13776.938339] LustreError: 436934:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13781.990539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13781.998899] LustreError: 436937:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13782.050929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13787.106443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13787.107448] LustreError: 436938:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13792.229896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13793.249211] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13793.587754] Lustre: server umount lustre-OST0000 complete [13797.347946] LustreError: 436937:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13797.356538] LustreError: 436937:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13804.227718] Lustre: server umount lustre-MDT0000 complete [13821.866222] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [13824.700756] Key type lgssc unregistered [13824.931731] LNet: 441693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13824.935264] LNetError: 441693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13824.958837] LNet: Removed LNI 192.168.201.126@tcp [13825.725069] Key type .llcrypt unregistered [13825.726712] Key type ._llcrypt unregistered [13874.739746] Key type ._llcrypt registered [13874.741505] Key type .llcrypt registered [13874.857041] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [13875.179581] Key type .llcrypt unregistered [13875.181805] Key type ._llcrypt unregistered [13892.453667] Key type ._llcrypt registered [13892.455074] Key type .llcrypt registered [13892.568209] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [13893.804877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13893.909636] alg: No test for adler32 (adler32-zlib) [13895.038874] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [13895.452802] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [13897.160223] Key type lgssc registered [13898.265581] Lustre: Echo OBD driver; http://www.lustre.org/ [13928.612788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13928.636036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13929.831983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13929.865185] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13929.945057] Lustre: lustre-MDT0000: new disk, initializing [13930.024817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13930.037689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13934.464434] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [13946.742594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13946.940534] Lustre: 445099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13947.036913] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13947.046240] Lustre: Skipped 1 previous similar message [13947.211558] Lustre: lustre-MDT0001: new disk, initializing [13947.479254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13947.558966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13947.589992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13953.792603] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [13965.460879] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13972.165980] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13979.577119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13979.803635] Lustre: lustre-OST0000: new disk, initializing [13979.810305] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13979.816597] Lustre: 447069:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13979.894143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13986.942564] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [13987.372332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13987.386378] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13987.455365] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13996.643650] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14003.691203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14003.700136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14003.716530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14007.778646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14007.779652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14009.930205] Lustre: server umount lustre-MDT0000 complete [14013.830500] LustreError: 445093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790793690 with bad export cookie 1924642908333169811 [14013.848509] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14014.277559] Lustre: server umount lustre-MDT0001 complete [14029.277765] Lustre: server umount lustre-OST0000 complete [14043.121670] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 14:41:58 (1790793718) [14052.613358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14053.098386] LustreError: 449099:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14053.128071] LustreError: 449099:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [14053.189785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14058.478577] LustreError: 449100:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14060.441446] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [14063.593183] LustreError: 449099:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14068.707811] LustreError: 449100:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14071.346436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14071.837683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14078.004831] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [14087.277183] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14094.243599] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14115.430962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14115.651361] Lustre: Found index 0 for lustre-OST0000, updating log [14115.913055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14121.738623] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [14123.074478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14130.233882] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14137.313739] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14137.329062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14137.346267] Lustre: Skipped 1 previous similar message [14137.350196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14137.354660] Lustre: Skipped 2 previous similar messages [14138.106643] Lustre: server umount lustre-OST0000 complete [14155.428614] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14155.444420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14155.456731] Lustre: Skipped 2 previous similar messages [14155.481728] Lustre: Skipped 2 previous similar messages [14160.917813] LustreError: 449099:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14161.115697] Lustre: server umount lustre-MDT0000 complete [14166.098153] LustreError: 449081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790793843 with bad export cookie 1924642908333171204 [14166.132130] LustreError: 449081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14166.158053] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14166.613801] Lustre: server umount lustre-MDT0001 complete [14177.681977] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 14:44:13 (1790793853) [14183.266081] LNet: 1 local NIs in recovery (showing 1): 192.168.201.126@tcp [14807.913763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14808.628214] LustreError: 455513:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14808.658256] LustreError: 455513:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [14808.744318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14813.551640] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [14814.178272] LustreError: 455514:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14818.279784] LustreError: 455513:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14822.872996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14823.628317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14829.740371] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [14839.953216] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14846.566361] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14862.304164] LNet: 1 local NIs in recovery (showing 1): 192.168.201.126@tcp [14867.314995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14867.605137] Lustre: Found index 0 for lustre-OST0000, updating log [14867.999152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14874.922670] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [14883.290679] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14886.370133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14886.386595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14886.418542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14891.671616] Lustre: server umount lustre-OST0000 complete [14896.681492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14896.686755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14896.688376] Lustre: Skipped 1 previous similar message [14896.694052] Lustre: Skipped 1 previous similar message [14897.697687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14899.749301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14899.761436] Lustre: Skipped 3 previous similar messages [14902.527301] Lustre: server umount lustre-MDT0000 complete [14902.908502] LustreError: 455513:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14902.950872] LustreError: 455513:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14907.079027] LustreError: 455493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790794584 with bad export cookie 1924642908333172289 [14907.081795] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14907.100853] LustreError: 455493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [14907.665570] Lustre: server umount lustre-MDT0001 complete [14922.724320] LNet: 1 local NIs in recovery (showing 1): 192.168.201.126@tcp [14943.773799] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [14956.451139] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [14992.851438] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 14:57:48 (1790794668) [15012.119732] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [15015.010379] Key type lgssc unregistered [15015.309979] LNet: 462131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15015.316933] LNetError: 462131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15127.976202] LNet: Removed LNI 192.168.201.126@tcp [15129.088715] Key type .llcrypt unregistered [15129.092282] Key type ._llcrypt unregistered [15150.963676] Key type ._llcrypt registered [15150.968118] Key type .llcrypt registered [15151.164482] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [15153.030072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15153.124262] alg: No test for adler32 (adler32-zlib) [15154.449568] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [15154.860138] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [15156.689467] Key type lgssc registered [15158.612797] Lustre: Echo OBD driver; http://www.lustre.org/ [15165.032867] LNet: Added LNI 192.168.201.126@tcp100 [8/256/0/180] [15165.987970] LNet: Added LNI 192.168.201.126@tcp101 [8/256/0/180] [15166.873562] LNet: Added LNI 192.168.201.126@tcp102 [8/256/0/180] [15167.692807] LNet: Added LNI 192.168.201.126@tcp103 [8/256/0/180] [15168.230759] LNet: Added LNI 192.168.201.126@tcp104 [8/256/0/180] [15168.909241] LNet: Added LNI 192.168.201.126@tcp105 [8/256/0/180] [15170.109411] LNet: Added LNI 192.168.201.126@tcp106 [8/256/0/180] [15171.274649] LNet: Added LNI 192.168.201.126@tcp107 [8/256/0/180] [15172.679313] LNet: Added LNI 192.168.201.126@tcp108 [8/256/0/180] [15173.629651] LNet: Added LNI 192.168.201.126@tcp109 [8/256/0/180] [15174.755024] LNet: Added LNI 192.168.201.126@tcp110 [8/256/0/180] [15175.678926] LNet: Added LNI 192.168.201.126@tcp111 [8/256/0/180] [15176.583928] LNet: Added LNI 192.168.201.126@tcp112 [8/256/0/180] [15177.732630] LNet: Added LNI 192.168.201.126@tcp113 [8/256/0/180] [15178.482713] LNet: Added LNI 192.168.201.126@tcp114 [8/256/0/180] [15179.727989] LNet: Added LNI 192.168.201.126@tcp115 [8/256/0/180] [15180.834815] LNet: Added LNI 192.168.201.126@tcp116 [8/256/0/180] [15181.943897] LNet: Added LNI 192.168.201.126@tcp117 [8/256/0/180] [15182.958145] LNet: Added LNI 192.168.201.126@tcp118 [8/256/0/180] [15184.070694] LNet: Added LNI 192.168.201.126@tcp119 [8/256/0/180] [15185.054789] LNet: Added LNI 192.168.201.126@tcp120 [8/256/0/180] [15186.018019] LNet: Added LNI 192.168.201.126@tcp121 [8/256/0/180] [15187.156617] LNet: Added LNI 192.168.201.126@tcp122 [8/256/0/180] [15188.495216] LNet: Added LNI 192.168.201.126@tcp123 [8/256/0/180] [15189.766281] LNet: Added LNI 192.168.201.126@tcp124 [8/256/0/180] [15190.408894] LNet: Added LNI 192.168.201.126@tcp125 [8/256/0/180] [15191.122348] LNet: Added LNI 192.168.201.126@tcp126 [8/256/0/180] [15191.890394] LNet: Added LNI 192.168.201.126@tcp127 [8/256/0/180] [15192.588534] LNet: Added LNI 192.168.201.126@tcp128 [8/256/0/180] [15193.793465] LNet: Added LNI 192.168.201.126@tcp129 [8/256/0/180] [15194.814816] LNet: Added LNI 192.168.201.126@tcp130 [8/256/0/180] [15195.753743] LNet: Added LNI 192.168.201.126@tcp131 [8/256/0/180] [15196.449566] LNet: Added LNI 192.168.201.126@tcp132 [8/256/0/180] [15197.479731] LNet: Added LNI 192.168.201.126@tcp133 [8/256/0/180] [15198.711240] LNet: Added LNI 192.168.201.126@tcp134 [8/256/0/180] [15199.878686] LNet: Added LNI 192.168.201.126@tcp135 [8/256/0/180] [15208.528062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [15208.547842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15209.661283] Lustre: 465190:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.126@tcp: can't add NID 192.168.201.126@tcp130: rc = -75 [15209.722266] LNetError: 462628:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.126 [15209.739416] LNetError: Protocol error connecting to 192.168.201.126@tcp120 at host 192.168.201.126:988: is it running a compatible version of Lustre? [15209.750687] LNetError: Skipped 1 previous similar message [15209.821348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15209.848420] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15209.919665] Lustre: lustre-MDT0000: new disk, initializing [15209.988610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15210.001298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15210.144306] LNet: 7 local NIs in recovery (showing 7): 192.168.201.126@tcp123, 192.168.201.126@tcp124, 192.168.201.126@tcp125, 192.168.201.126@tcp126, 192.168.201.126@tcp120, 192.168.201.126@tcp128, 192.168.201.126@tcp127 [15212.196962] LNetError: 462629:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.126 [15212.212784] LNetError: 462629:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 18 previous similar messages [15212.227334] LNetError: Protocol error connecting to 192.168.201.126@tcp123 at host 192.168.201.126:988: is it running a compatible version of Lustre? [15212.235558] LNetError: Skipped 14 previous similar messages [15214.272545] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [15216.289371] LNetError: 462628:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.126 [15216.309260] LNetError: 462628:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [15216.312595] LNetError: Protocol error connecting to 192.168.201.126@tcp123 at host 192.168.201.126:988: is it running a compatible version of Lustre? [15216.321529] LNetError: Skipped 1 previous similar message [15224.482629] LNetError: 465215:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.126 [15224.490801] LNetError: Protocol error connecting to 192.168.201.126@tcp123 at host 192.168.201.126:988: is it running a compatible version of Lustre? [15225.168409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15225.307944] Lustre: 465207:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15225.367930] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15225.372861] Lustre: Skipped 1 previous similar message [15225.465591] Lustre: lustre-MDT0001: new disk, initializing [15225.498168] Lustre: 466006:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.126@tcp: can't add NID 192.168.201.126@tcp130: rc = -75 [15225.510663] Lustre: 466006:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15225.625810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15225.667850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15225.684037] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15229.734789] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [15233.105369] Lustre: 466390:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.126@tcp: can't add NID 192.168.201.126@tcp130: rc = -75 [15233.111426] Lustre: 466390:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15240.530981] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15240.867676] LNetError: 462626:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.126 [15240.888660] LNetError: Protocol error connecting to 192.168.201.126@tcp123 at host 192.168.201.126:988: is it running a compatible version of Lustre? [15248.225849] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15249.124743] LNetError: 462629:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.126 [15249.133848] LNetError: 462629:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [15249.142619] LNetError: Protocol error connecting to 192.168.201.126@tcp121 at host 192.168.201.126:988: is it running a compatible version of Lustre? [15249.157203] LNetError: Skipped 4 previous similar messages [15256.665234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15257.124697] Lustre: lustre-OST0000: new disk, initializing [15257.130088] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15257.137707] Lustre: 467182:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15257.235495] Lustre: 467187:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.126@tcp: can't add NID 192.168.201.126@tcp130: rc = -75 [15257.239949] Lustre: 467187:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15257.305280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15262.747807] Lustre: 467441:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.126@tcp: can't add NID 192.168.201.126@tcp130: rc = -75 [15262.769090] Lustre: 467441:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15262.869330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15262.883952] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15263.087282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15264.922975] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [15266.530936] LNetError: 462628:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.126 [15266.548162] LNetError: 462628:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [15266.556948] LNetError: Protocol error connecting to 192.168.201.126@tcp122 at host 192.168.201.126:988: is it running a compatible version of Lustre? [15266.569887] LNetError: Skipped 9 previous similar messages [15270.568954] LNet: 1 local NIs in recovery (showing 1): 192.168.201.126@tcp123 [15275.814967] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15288.818197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15288.819348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15288.821487] LustreError: Skipped 1 previous similar message [15288.823582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15288.832680] Lustre: Skipped 1 previous similar message [15288.880657] Lustre: Skipped 1 previous similar message [15291.922123] Lustre: server umount lustre-OST0000 complete [15297.513901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15297.526840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15297.557231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15299.045027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15299.047748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15304.166648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15304.169274] Lustre: Skipped 1 previous similar message [15305.446579] LNetError: 465222:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.126 [15305.453643] LNetError: 465222:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 14 previous similar messages [15305.469178] LNetError: Protocol error connecting to 192.168.201.126@tcp122 at host 192.168.201.126:988: is it running a compatible version of Lustre? [15305.480040] LNetError: Skipped 14 previous similar messages [15308.271579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15308.277877] Lustre: Skipped 1 previous similar message [15311.329435] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15311.588743] Lustre: server umount lustre-MDT0000 complete [15317.119115] LustreError: 465200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790794994 with bad export cookie 6006801829429645842 [15317.124559] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15317.142643] LustreError: 465200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15317.750554] Lustre: server umount lustre-MDT0001 complete [15331.395428] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [15334.915752] Key type lgssc unregistered [15335.553216] LNet: 469098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15335.568097] LNetError: 469098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15335.589757] LNet: Removed LNI 192.168.201.126@tcp [15336.608166] LNet: Removed LNI 192.168.201.126@tcp100 [15336.609988] LNetError: 469098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15336.616979] LNetError: 469098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15336.629335] LNet: Removed LNI 192.168.201.126@tcp101 [15337.696401] LNet: Removed LNI 192.168.201.126@tcp102 [15337.698441] LNetError: 469098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15337.709963] LNetError: 469098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15337.719067] LNet: Removed LNI 192.168.201.126@tcp103 [15338.784201] LNet: Removed LNI 192.168.201.126@tcp104 [15339.808210] LNet: Removed LNI 192.168.201.126@tcp105 [15339.833146] LNetError: 469098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15339.845988] LNetError: 469098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15340.896389] LNet: Removed LNI 192.168.201.126@tcp106 [15341.920272] LNet: Removed LNI 192.168.201.126@tcp107 [15342.944177] LNet: Removed LNI 192.168.201.126@tcp108 [15343.969993] LNet: Removed LNI 192.168.201.126@tcp109 [15343.979168] LNetError: 469098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15344.002552] LNetError: 469098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [15344.037085] LNet: Removed LNI 192.168.201.126@tcp110 [15345.056742] LNet: Removed LNI 192.168.201.126@tcp111 [15346.080192] LNet: Removed LNI 192.168.201.126@tcp112 [15347.106314] LNet: Removed LNI 192.168.201.126@tcp113 [15348.128656] LNet: Removed LNI 192.168.201.126@tcp114 [15349.152307] LNet: Removed LNI 192.168.201.126@tcp115 [15350.176378] LNet: Removed LNI 192.168.201.126@tcp116 [15351.203835] LNet: Removed LNI 192.168.201.126@tcp117 [15352.224304] LNet: Removed LNI 192.168.201.126@tcp118 [15352.226217] LNetError: 469098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15352.241961] LNetError: 469098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15352.254764] LNet: Removed LNI 192.168.201.126@tcp119 [15353.320424] LNet: Removed LNI 192.168.201.126@tcp120 [15354.336675] LNet: Removed LNI 192.168.201.126@tcp121 [15355.360435] LNet: Removed LNI 192.168.201.126@tcp122 [15355.362611] LNet: Removed LNI 192.168.201.126@tcp123 [15356.384485] LNet: Removed LNI 192.168.201.126@tcp124 [15357.410328] LNet: Removed LNI 192.168.201.126@tcp125 [15358.433986] LNet: Removed LNI 192.168.201.126@tcp126 [15359.456461] LNet: Removed LNI 192.168.201.126@tcp127 [15360.480213] LNet: Removed LNI 192.168.201.126@tcp128 [15361.504420] LNet: Removed LNI 192.168.201.126@tcp129 [15361.507801] LNet: Removed LNI 192.168.201.126@tcp130 [15361.509653] LNet: Removed LNI 192.168.201.126@tcp131 [15361.517735] LNet: Removed LNI 192.168.201.126@tcp132 [15361.523313] LNet: Removed LNI 192.168.201.126@tcp133 [15361.528704] LNet: Removed LNI 192.168.201.126@tcp134 [15361.547073] LNet: Removed LNI 192.168.201.126@tcp135 [15362.556611] Key type .llcrypt unregistered [15362.559129] Key type ._llcrypt unregistered [15391.626215] Key type ._llcrypt registered [15391.628247] Key type .llcrypt registered [15391.717896] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [15414.227038] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [15415.910841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15415.992854] alg: No test for adler32 (adler32-zlib) [15417.250497] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [15417.672988] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [15419.336224] Key type lgssc registered [15421.395354] Lustre: Echo OBD driver; http://www.lustre.org/ [15462.785372] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 15:05:38 (1790795138) [15483.992306] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [15487.667823] Key type lgssc unregistered [15488.068230] LNet: 473206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15488.075602] LNetError: 473206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15488.102984] LNet: Removed LNI 192.168.201.126@tcp [15489.160300] Key type .llcrypt unregistered [15489.162529] Key type ._llcrypt unregistered [15509.185290] Key type ._llcrypt registered [15509.187531] Key type .llcrypt registered [15509.302628] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [15510.648958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15510.666525] alg: No test for adler32 (adler32-zlib) [15511.695851] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [15511.990478] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [15513.696277] Key type lgssc registered [15515.090743] Lustre: Echo OBD driver; http://www.lustre.org/ [15525.877393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [15525.934828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15527.162264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15527.189880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15527.296929] Lustre: lustre-MDT0000: new disk, initializing [15527.427910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15527.448357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15532.967565] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [15544.366186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15544.585470] Lustre: 474434:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15544.661319] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15544.665496] Lustre: Skipped 1 previous similar message [15544.768390] Lustre: lustre-MDT0001: new disk, initializing [15544.910612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15545.009566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15545.032317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15551.114930] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [15562.468840] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15569.954727] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15579.290704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15579.698475] Lustre: lustre-OST0000: new disk, initializing [15579.714373] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15579.725540] Lustre: 476406:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15579.881927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15585.861397] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [15586.928138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15586.937858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15587.117203] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15597.022988] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15601.721119] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing lnet_if_list [15608.591073] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15608.632343] LNet: Added LNI 192.168.201.126@tcp10 [8/256/0/180] [15614.890394] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15614.923267] LNet: Added LNI 192.168.201.126@tcp11 [8/256/0/180] [15619.732248] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15619.755709] LNet: Added LNI 192.168.201.126@tcp12 [8/256/0/180] [15625.041820] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15625.091391] LNet: Added LNI 192.168.201.126@tcp13 [8/256/0/180] [15630.257693] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15630.291247] LNet: Added LNI 192.168.201.126@tcp14 [8/256/0/180] [15635.831824] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15635.850336] LNet: Added LNI 192.168.201.126@tcp15 [8/256/0/180] [15642.429867] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15642.456864] LNet: Added LNI 192.168.201.126@tcp16 [8/256/0/180] [15648.520447] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15648.559250] LNet: Added LNI 192.168.201.126@tcp17 [8/256/0/180] [15653.839640] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15653.865188] LNet: Added LNI 192.168.201.126@tcp18 [8/256/0/180] [15658.615890] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15658.654147] LNet: Added LNI 192.168.201.126@tcp19 [8/256/0/180] [15663.731734] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15663.766894] LNet: Added LNI 192.168.201.126@tcp20 [8/256/0/180] [15670.178243] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15670.205852] LNet: Added LNI 192.168.201.126@tcp21 [8/256/0/180] [15675.694284] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15675.736725] LNet: Added LNI 192.168.201.126@tcp22 [8/256/0/180] [15681.636384] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15681.671303] LNet: Added LNI 192.168.201.126@tcp23 [8/256/0/180] [15687.158149] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15687.187348] LNet: Added LNI 192.168.201.126@tcp24 [8/256/0/180] [15693.558576] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15693.592974] LNet: Added LNI 192.168.201.126@tcp25 [8/256/0/180] [15701.092260] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15701.126720] LNet: Added LNI 192.168.201.126@tcp26 [8/256/0/180] [15707.445571] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15707.473260] LNet: Added LNI 192.168.201.126@tcp27 [8/256/0/180] [15714.759288] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15714.800277] LNet: Added LNI 192.168.201.126@tcp28 [8/256/0/180] [15721.831784] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15721.874537] LNet: Added LNI 192.168.201.126@tcp29 [8/256/0/180] [15728.909913] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15728.950137] LNet: Added LNI 192.168.201.126@tcp30 [8/256/0/180] [15736.279958] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15736.330710] LNet: Added LNI 192.168.201.126@tcp31 [8/256/0/180] [15744.143395] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15744.168069] LNet: Added LNI 192.168.201.126@tcp32 [8/256/0/180] [15751.945462] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15752.005922] LNet: Added LNI 192.168.201.126@tcp33 [8/256/0/180] [15759.078952] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15759.141773] LNet: Added LNI 192.168.201.126@tcp34 [8/256/0/180] [15766.731779] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15766.796362] LNet: Added LNI 192.168.201.126@tcp35 [8/256/0/180] [15774.535271] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15774.564613] LNet: Added LNI 192.168.201.126@tcp36 [8/256/0/180] [15781.656535] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15781.699753] LNet: Added LNI 192.168.201.126@tcp37 [8/256/0/180] [15789.763036] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15789.801562] LNet: Added LNI 192.168.201.126@tcp38 [8/256/0/180] [15798.067405] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15798.108523] LNet: Added LNI 192.168.201.126@tcp39 [8/256/0/180] [15808.190985] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15808.253881] LNet: Added LNI 192.168.201.126@tcp40 [8/256/0/180] [15817.583636] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15817.630840] LNet: Added LNI 192.168.201.126@tcp41 [8/256/0/180] [15825.773604] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15825.805089] LNet: Added LNI 192.168.201.126@tcp42 [8/256/0/180] [15833.905186] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15833.952566] LNet: Added LNI 192.168.201.126@tcp43 [8/256/0/180] [15842.952906] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15843.012292] LNet: Added LNI 192.168.201.126@tcp44 [8/256/0/180] [15852.525678] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15852.556216] LNet: Added LNI 192.168.201.126@tcp45 [8/256/0/180] [15861.678449] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15861.708065] LNet: Added LNI 192.168.201.126@tcp46 [8/256/0/180] [15872.097399] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15872.139254] LNet: Added LNI 192.168.201.126@tcp47 [8/256/0/180] [15883.117714] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15883.188410] LNet: Added LNI 192.168.201.126@tcp48 [8/256/0/180] [15893.896610] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15893.934365] LNet: Added LNI 192.168.201.126@tcp49 [8/256/0/180] [15905.911346] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15905.946241] LNet: Added LNI 192.168.201.126@tcp50 [8/256/0/180] [15928.802689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15928.817554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15928.831849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15930.341019] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15930.345490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15930.368698] Lustre: Skipped 1 previous similar message [15930.374198] Lustre: Skipped 1 previous similar message [15934.720452] Lustre: server umount lustre-MDT0000 complete [15939.502833] LustreError: 474429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790795616 with bad export cookie 10884937807737957491 [15939.515599] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15939.519697] LustreError: 474429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15939.886205] Lustre: server umount lustre-MDT0001 complete [15956.960232] Lustre: 473699:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790795617/real 1790795617] req@0000000013e329cc x1877784856283776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790795633 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15957.012096] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15960.096769] Lustre: 473699:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790795621/real 1790795621] req@0000000026781270 x1877784856283904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790795637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15964.920698] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [15965.152414] Lustre: 473698:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790795626/real 1790795626] req@00000000114a49d2 x1877784856284288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790795642 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15965.208405] Lustre: 473698:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15970.285782] Lustre: 473700:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790795631/real 1790795631] req@00000000972321f0 x1877784856284416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790795647 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15980.508356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15985.081551] LustreError: 476400:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15985.117837] LustreError: 476400:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [15985.185516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15986.218589] LustreError: 476401:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15986.246382] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15987.247503] LustreError: 476400:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15990.744724] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [15992.299245] LustreError: 476401:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15997.423914] LustreError: 476400:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16001.488272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16002.061425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16003.068901] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16007.941269] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [16018.689571] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16025.007094] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16049.123053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16049.131931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16049.136946] Lustre: Skipped 1 previous similar message [16049.154761] Lustre: Skipped 2 previous similar messages [16054.265958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16054.280436] Lustre: Skipped 1 previous similar message [16055.036374] Lustre: server umount lustre-OST0000 complete [16063.457648] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16063.464780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16063.476905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16065.728141] Lustre: server umount lustre-MDT0000 complete [16069.423232] LustreError: 474429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790795746 with bad export cookie 10884937807737960018 [16069.440877] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16069.793439] Lustre: server umount lustre-MDT0001 complete [16084.434282] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [16087.256580] Key type lgssc unregistered [16087.771539] LNet: 503452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16087.779849] LNetError: 503452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16087.788432] LNet: Removed LNI 192.168.201.126@tcp [16087.791805] LNet: Removed LNI 192.168.201.126@tcp10 [16087.804955] LNet: Removed LNI 192.168.201.126@tcp11 [16087.809185] LNet: Removed LNI 192.168.201.126@tcp12 [16087.813581] LNet: Removed LNI 192.168.201.126@tcp13 [16087.817364] LNet: Removed LNI 192.168.201.126@tcp14 [16087.820596] LNet: Removed LNI 192.168.201.126@tcp15 [16087.824734] LNet: Removed LNI 192.168.201.126@tcp16 [16087.832380] LNet: Removed LNI 192.168.201.126@tcp17 [16087.836607] LNet: Removed LNI 192.168.201.126@tcp18 [16087.839862] LNet: Removed LNI 192.168.201.126@tcp19 [16087.842510] LNet: Removed LNI 192.168.201.126@tcp20 [16087.849239] LNet: Removed LNI 192.168.201.126@tcp21 [16087.854982] LNet: Removed LNI 192.168.201.126@tcp22 [16087.860974] LNet: Removed LNI 192.168.201.126@tcp23 [16087.867365] LNet: Removed LNI 192.168.201.126@tcp24 [16087.871413] LNet: Removed LNI 192.168.201.126@tcp25 [16087.874712] LNet: Removed LNI 192.168.201.126@tcp26 [16087.878538] LNet: Removed LNI 192.168.201.126@tcp27 [16087.880338] LNet: Removed LNI 192.168.201.126@tcp28 [16087.883558] LNet: Removed LNI 192.168.201.126@tcp29 [16087.886451] LNet: Removed LNI 192.168.201.126@tcp30 [16087.889189] LNet: Removed LNI 192.168.201.126@tcp31 [16087.892386] LNet: Removed LNI 192.168.201.126@tcp32 [16087.895241] LNet: Removed LNI 192.168.201.126@tcp33 [16087.898114] LNet: Removed LNI 192.168.201.126@tcp34 [16087.900766] LNet: Removed LNI 192.168.201.126@tcp35 [16087.902013] LNet: Removed LNI 192.168.201.126@tcp36 [16087.914529] LNet: Removed LNI 192.168.201.126@tcp37 [16087.917211] LNet: Removed LNI 192.168.201.126@tcp38 [16087.922399] LNet: Removed LNI 192.168.201.126@tcp39 [16087.925903] LNet: Removed LNI 192.168.201.126@tcp40 [16087.928334] LNet: Removed LNI 192.168.201.126@tcp41 [16087.930709] LNet: Removed LNI 192.168.201.126@tcp42 [16087.934102] LNet: Removed LNI 192.168.201.126@tcp43 [16087.937269] LNet: Removed LNI 192.168.201.126@tcp44 [16087.940790] LNet: Removed LNI 192.168.201.126@tcp45 [16087.943635] LNet: Removed LNI 192.168.201.126@tcp46 [16087.947223] LNet: Removed LNI 192.168.201.126@tcp47 [16087.951432] LNet: Removed LNI 192.168.201.126@tcp48 [16087.956155] LNet: Removed LNI 192.168.201.126@tcp49 [16087.965040] LNet: Removed LNI 192.168.201.126@tcp50 [16089.035646] Key type .llcrypt unregistered [16089.038120] Key type ._llcrypt unregistered [16122.066506] Key type ._llcrypt registered [16122.068789] Key type .llcrypt registered [16122.359678] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [16152.624688] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [16155.920795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16155.982345] alg: No test for adler32 (adler32-zlib) [16157.298960] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [16157.691284] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [16159.408192] Key type lgssc registered [16160.478699] Lustre: Echo OBD driver; http://www.lustre.org/ [16199.541772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [16199.591788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16200.853893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16200.883314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16200.983348] Lustre: lustre-MDT0000: new disk, initializing [16201.090957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16201.114504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16206.491691] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [16217.892593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16218.118095] Lustre: 506668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16218.225824] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16218.233120] Lustre: Skipped 1 previous similar message [16218.356200] Lustre: lustre-MDT0001: new disk, initializing [16218.523996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16218.592079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16218.606273] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16223.331696] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [16234.343254] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16240.956736] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16248.061963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16248.236206] Lustre: *** cfs_fail_loc=90f, val=0*** [16248.287701] Lustre: lustre-OST0000: new disk, initializing [16248.295078] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16248.299995] Lustre: 508685:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16248.376342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16254.703280] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [16257.090047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16257.097240] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16257.269747] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16265.895689] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16277.997755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16278.001054] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16278.001486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16278.011315] Lustre: Skipped 1 previous similar message [16280.672957] Lustre: server umount lustre-OST0000 complete [16285.158891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16285.166917] LustreError: Skipped 1 previous similar message [16285.175693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16285.185752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16285.188276] Lustre: Skipped 1 previous similar message [16288.250243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16288.255621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16291.151236] Lustre: server umount lustre-MDT0000 complete [16295.693791] LustreError: 507465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790795972 with bad export cookie 7258247412360768475 [16295.704398] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16295.711945] LustreError: 507465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16296.179482] Lustre: server umount lustre-MDT0001 complete [16308.505312] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [16311.381834] Key type lgssc unregistered [16311.654548] LNet: 510593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16311.668767] LNetError: 510593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16311.692772] LNet: Removed LNI 192.168.201.126@tcp [16312.515173] Key type .llcrypt unregistered [16312.517104] Key type ._llcrypt unregistered [16334.990588] Key type ._llcrypt registered [16334.999198] Key type .llcrypt registered [16335.123907] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [16337.227895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16337.408826] alg: No test for adler32 (adler32-zlib) [16338.672448] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [16339.074611] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [16340.920722] Key type lgssc registered [16342.233759] Lustre: Echo OBD driver; http://www.lustre.org/ [16353.308978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [16353.327478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16354.741133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16360.380080] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [16364.008413] LustreError: 511881:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16364.039958] LustreError: 511881:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [16369.128322] LustreError: 511882:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16369.753894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16370.055720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16374.765551] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [16382.815827] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16389.754672] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16397.071422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16397.468048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16404.744627] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [16413.884693] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16427.563618] loop0: detected capacity change from 0 to 204800000 [16427.792447] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16427.826809] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16427.851085] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16436.534888] loop0: detected capacity change from 0 to 204800000 [16436.989965] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16437.048623] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16437.075744] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16444.589423] loop0: detected capacity change from 0 to 204800000 [16454.329462] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16454.517247] Lustre: fs15246-OST0000: new disk, initializing [16454.522508] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16454.527610] Lustre: 515476:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16454.603725] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16464.329267] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [16476.464331] loop1: detected capacity change from 0 to 204800000 [16483.456569] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16483.635226] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16483.680565] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16483.750349] Lustre: fs15246-MDT0000: new disk, initializing [16483.903212] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16483.958330] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16484.150971] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16485.220540] Lustre: 516601:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000bdabe01d x1877785723325312/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790796178 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16489.464203] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [16494.319988] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16494.330541] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16494.361185] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16516.080642] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16516.095449] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16517.376172] Lustre: server umount fs15246-MDT0000 complete [16521.197089] LustreError: 514075:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16526.313889] LustreError: 517922:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16527.770896] Lustre: server umount fs15246-OST0000 complete [16541.676177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16541.693294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16541.696070] Lustre: Skipped 1 previous similar message [16541.712538] Lustre: Skipped 1 previous similar message [16546.477322] Lustre: server umount lustre-OST0000 complete [16551.906630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16551.909581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16551.926958] Lustre: Skipped 1 previous similar message [16551.947598] Lustre: Skipped 1 previous similar message [16556.995372] Lustre: server umount lustre-MDT0000 complete [16557.030310] LustreError: 511881:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16557.056895] LustreError: 511881:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16561.676392] LustreError: 511862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790796238 with bad export cookie 9009250823433157731 [16561.709173] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16562.225681] Lustre: server umount lustre-MDT0001 complete [16574.616238] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [16578.678459] Key type lgssc unregistered [16579.275371] LNet: 519698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16579.306190] LNetError: 519698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16579.361943] LNet: Removed LNI 192.168.201.126@tcp [16581.189248] Key type .llcrypt unregistered [16581.196734] Key type ._llcrypt unregistered [16606.516620] Key type ._llcrypt registered [16606.522812] Key type .llcrypt registered [16606.632470] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [16607.967481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16608.039140] alg: No test for adler32 (adler32-zlib) [16609.189495] Lustre: Lustre: Build Version: 2.17.59_2_g696972c [16609.491219] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [16611.208191] Key type lgssc registered [16613.357815] Lustre: Echo OBD driver; http://www.lustre.org/ [16661.237709] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [16673.429845] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [16718.676918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [16718.717996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16720.053666] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16720.075389] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16720.216350] Lustre: lustre-MDT0000: new disk, initializing [16720.290958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16720.325353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16724.798304] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [16739.675057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16739.940478] Lustre: 524194:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16740.132948] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16740.148231] Lustre: Skipped 1 previous similar message [16740.289087] Lustre: lustre-MDT0001: new disk, initializing [16740.389441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16740.443868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16740.453056] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16746.518762] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [16757.659700] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16763.909257] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16771.423642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16771.764671] Lustre: lustre-OST0000: new disk, initializing [16771.779783] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16771.797633] Lustre: 526167:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16771.901608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16778.944758] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [16779.941191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16779.953435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16780.091170] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16788.444173] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16793.145230] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16793.309757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16797.767681] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16797.945572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16800.740169] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16800.745612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16800.768146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16800.778490] Lustre: Skipped 1 previous similar message [16800.797345] Lustre: Skipped 1 previous similar message [16806.111781] Lustre: server umount lustre-OST0000 complete [16810.979238] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16810.984707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16811.002119] Lustre: Skipped 1 previous similar message [16816.101939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16816.112445] Lustre: Skipped 2 previous similar messages [16821.217209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16821.222954] Lustre: Skipped 1 previous similar message [16824.288144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16826.343958] LustreError: 525862:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16826.362827] LustreError: 525862:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [16826.518704] Lustre: server umount lustre-MDT0000 complete [16830.560770] LustreError: 524186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790796507 with bad export cookie 1953329568972849336 [16830.571471] LustreError: 524186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16830.580023] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16830.925716] Lustre: server umount lustre-MDT0001 complete [16857.075544] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [16868.699549] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [16903.226437] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16579 sec ========= 15:29:39 (1790796579) [16905.120517] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:29:40 (1790796580) === [16908.806840] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:29:44 (1790796584) === [16952.482551] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [16955.509768] Key type lgssc unregistered [16955.811409] LNet: 532814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16955.819786] LNetError: 532814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16955.842132] LNet: Removed LNI 192.168.201.126@tcp [16956.788778] Key type .llcrypt unregistered [16956.790598] Key type ._llcrypt unregistered